[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:00 dist-test-slave-q9h9 krb5kdc[28405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:00 dist-test-slave-q9h9 krb5kdc[28405](info): set up 2 sockets
Aug 09 15:01:00 dist-test-slave-q9h9 krb5kdc[28405](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 15:01:01.977911 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:02 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:02.148051 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42175
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:42175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:02.479039 28422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:02.479324 28422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:02.479462 28422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:02.486459 28422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:02.486562 28422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:02.486642 28422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:02.486725 28422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:02.486788 28422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:02.496490 28422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:42175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:02.497793 28422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:02.499125 28422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:02.515213 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:02.515709 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:02.516040 28432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:02.516537 28422 server_base.cc:1034] running on GCE node
I20260809 15:01:02.517316 28422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:02.518586 28422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:02.519804 28422 hybrid_clock.cc:648] HybridClock initialized: now 1786287662519759 us; error 71 us; skew 500 ppm
I20260809 15:01:02.520207 28422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:02 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287662, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:02.523193 28422 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:02.524768 28422 webserver.cc:460] Webserver started at http://127.27.186.190:34771/ using document root <none> and password file <none>
I20260809 15:01:02.525391 28422 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:02.525503 28422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:02.525775 28422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:02.529114 28422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72"
format_stamp: "Formatted at 2026-08-09 15:01:02 on dist-test-slave-q9h9"
server_key: "e1eb7dade8db07850659cf0686021029"
server_key_iv: "ec59fb6e959ff60da99f71dde6ee8cdc"
server_key_version: "encryptionkey@0"
I20260809 15:01:02.530301 28422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72"
format_stamp: "Formatted at 2026-08-09 15:01:02 on dist-test-slave-q9h9"
server_key: "e1eb7dade8db07850659cf0686021029"
server_key_iv: "ec59fb6e959ff60da99f71dde6ee8cdc"
server_key_version: "encryptionkey@0"
I20260809 15:01:02.535594 28422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 15:01:02.539225 28442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:02.540457 28422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 15:01:02.540673 28422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72"
format_stamp: "Formatted at 2026-08-09 15:01:02 on dist-test-slave-q9h9"
server_key: "e1eb7dade8db07850659cf0686021029"
server_key_iv: "ec59fb6e959ff60da99f71dde6ee8cdc"
server_key_version: "encryptionkey@0"
I20260809 15:01:02.540884 28422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:02.560026 28422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:02.563827 28422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:02.564062 28422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:02.583217 28422 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:42175
I20260809 15:01:02.583230 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:42175 every 8 connection(s)
I20260809 15:01:02.584467 28422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:02.588181 28521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:02.591274 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28422
I20260809 15:01:02.591538 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:02.593449 28394 external_mini_cluster.cc:1428] Setting key cbc15787c2f12daf2c73e52cac283a03
I20260809 15:01:02.598439 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72: Bootstrap starting.
I20260809 15:01:02.601213 28521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:02.602193 28521 log.cc:824] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:02.605372 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72: No bootstrap required, opened a new log
I20260809 15:01:02.616386 28521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42175 } }
I20260809 15:01:02.616977 28521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:02.617180 28521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984cdd6bf4d34cc5b37a4efb1e90ea72, State: Initialized, Role: FOLLOWER
I20260809 15:01:02.618060 28521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42175 } }
I20260809 15:01:02.618356 28521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:02.618505 28521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:02.618692 28521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:02.621085 28521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42175 } }
I20260809 15:01:02.621598 28521 leader_election.cc:302] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 984cdd6bf4d34cc5b37a4efb1e90ea72; no voters:
I20260809 15:01:02.623648 28521 leader_election.cc:288] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:02.623745 28533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:02.626830 28521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:02.626962 28533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [term 1 LEADER]: Becoming Leader. State: Replica: 984cdd6bf4d34cc5b37a4efb1e90ea72, State: Running, Role: LEADER
I20260809 15:01:02.627509 28533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42175 } }
Aug 09 15:01:02 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:02.641098 28535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 984cdd6bf4d34cc5b37a4efb1e90ea72. Latest consensus state: current_term: 1 leader_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42175 } } }
I20260809 15:01:02.641819 28535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:02.641503 28534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "984cdd6bf4d34cc5b37a4efb1e90ea72" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42175 } } }
I20260809 15:01:02.642683 28534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:02.645596 28548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:02.645905 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:02.597135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.27.186.190:42175)
0809 15:01:02.598057 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:02.598071 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:02.599344 (+ 1273us) server_negotiation.cc:407] Connection header received
0809 15:01:02.611509 (+ 12165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:02.611535 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:02.611983 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:02.612390 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:02.616986 (+ 4596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:02.619625 (+ 2639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:02.625006 (+ 5381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:02.626185 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:02.626417 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:02.633839 (+ 7422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:02.633851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:02.633870 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:02.633917 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:02.639928 (+ 6011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:02.640838 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:02.640847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:02.640855 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:02.641121 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:02.641653 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:02.641658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:02.641661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:02.642439 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:02.642754 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:02.643789 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0809 15:01:02.644205 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":444,"threads_started":1}
I20260809 15:01:02.650825 28548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:02.674429 28548 catalog_manager.cc:1347] Generated new cluster ID: 8b58a8fc530247b399b6d18bdc9bd013
I20260809 15:01:02.674527 28548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:02.688123 28548 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:02.689755 28548 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:02.696823 28548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 984cdd6bf4d34cc5b37a4efb1e90ea72: Generated new TSK 0
I20260809 15:01:02.697647 28548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:02.759379 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42175
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:03.089638 28561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:03.089880 28561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:03.089972 28561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:03.096868 28561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:03.096956 28561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:03.097069 28561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:03.106431 28561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:03.107465 28561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:03.108624 28561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:03.120771 28576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:03.121245 28578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:03.121417 28575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:03.122141 28561 server_base.cc:1034] running on GCE node
I20260809 15:01:03.122707 28561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:03.123508 28561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:03.124711 28561 hybrid_clock.cc:648] HybridClock initialized: now 1786287663124651 us; error 43 us; skew 500 ppm
I20260809 15:01:03.124987 28561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:03 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287663, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:03.128069 28561 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:03.129467 28561 webserver.cc:460] Webserver started at http://127.27.186.129:36973/ using document root <none> and password file <none>
I20260809 15:01:03.130127 28561 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:03.130242 28561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:03.130527 28561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:03.133467 28561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "81b96628d149417790c7b25292e83f0e"
format_stamp: "Formatted at 2026-08-09 15:01:03 on dist-test-slave-q9h9"
server_key: "cda62301a2db57ed55d1b59089c320e8"
server_key_iv: "4db4041b02278f3957dd5c859266dca2"
server_key_version: "encryptionkey@0"
I20260809 15:01:03.134212 28561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "81b96628d149417790c7b25292e83f0e"
format_stamp: "Formatted at 2026-08-09 15:01:03 on dist-test-slave-q9h9"
server_key: "cda62301a2db57ed55d1b59089c320e8"
server_key_iv: "4db4041b02278f3957dd5c859266dca2"
server_key_version: "encryptionkey@0"
I20260809 15:01:03.139237 28561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 15:01:03.142527 28585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:03.143746 28561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:03.143887 28561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "81b96628d149417790c7b25292e83f0e"
format_stamp: "Formatted at 2026-08-09 15:01:03 on dist-test-slave-q9h9"
server_key: "cda62301a2db57ed55d1b59089c320e8"
server_key_iv: "4db4041b02278f3957dd5c859266dca2"
server_key_version: "encryptionkey@0"
I20260809 15:01:03.144069 28561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:03.169872 28561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:03.173838 28561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:03.174134 28561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:03.175069 28561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:03.176296 28561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:03.176374 28561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:03.176486 28561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:03.176540 28561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:03.209340 28561 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:33511
I20260809 15:01:03.209398 28756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:33511 every 8 connection(s)
I20260809 15:01:03.210482 28561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:03.213878 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28561
I20260809 15:01:03.214141 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:03.214890 28394 external_mini_cluster.cc:1428] Setting key e78c092b88f17dc77ffb9fbaa3e90ac2
Aug 09 15:01:03 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287663, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:03.230288 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:03.212949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:39785 (local address 127.27.186.190:42175)
0809 15:01:03.213346 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:03.213350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:03.214355 (+ 1005us) server_negotiation.cc:407] Connection header received
0809 15:01:03.216356 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:03.216361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:03.216436 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:03.216524 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:03.218075 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.219125 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.220584 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.221129 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.221257 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:03.224534 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:03.224539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:03.224542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:03.224572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:03.227320 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:03.228026 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:03.228031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:03.228034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:03.228166 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:03.228576 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:03.228581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:03.228585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:03.228853 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:03.228988 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:03.229653 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 15:01:03.229884 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":209,"threads_started":1}
I20260809 15:01:03.231164 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:03.213347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42175 (local address 127.27.186.129:39785)
0809 15:01:03.214191 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:03.214235 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:03.216091 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:03.216780 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:03.216791 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:03.217361 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:03.217880 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:03.217897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.219265 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.219269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:03.220438 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:03.220455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.221320 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.221325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:03.221508 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:03.222100 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:03.222120 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:03.224336 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:03.227459 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:03.227467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:03.227482 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:03.227917 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:03.228277 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:03.228282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:03.228286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:03.228471 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:03.228979 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:03.228987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:03.229297 (+ 310us) client_negotiation.cc:769] Sending connection context
0809 15:01:03.229534 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 15:01:03.230069 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":413,"threads_started":1}
I20260809 15:01:03.232786 28757 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42175
I20260809 15:01:03.233120 28757 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
I20260809 15:01:03.233999 28757 heartbeater.cc:511] Master 127.27.186.190:42175 requested a full tablet report, sending...
I20260809 15:01:03.236076 28469 ts_manager.cc:194] Registered new tserver with Master: 81b96628d149417790c7b25292e83f0e (127.27.186.129:33511)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
I20260809 15:01:03.237596 28469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:39785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:03.255213 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42175
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:03.565403 28766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:03.565632 28766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:03.565721 28766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:03.572513 28766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:03.572600 28766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:03.572717 28766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:03.581640 28766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:03.582875 28766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:03.584503 28766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:03.597965 28776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:03.597998 28777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:03.599052 28766 server_base.cc:1034] running on GCE node
W20260809 15:01:03.600643 28779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:03.601251 28766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:03.602103 28766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:03.603308 28766 hybrid_clock.cc:648] HybridClock initialized: now 1786287663603263 us; error 36 us; skew 500 ppm
I20260809 15:01:03.603585 28766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:03 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287663, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:03.606478 28766 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:03.607832 28766 webserver.cc:460] Webserver started at http://127.27.186.130:34637/ using document root <none> and password file <none>
I20260809 15:01:03.608395 28766 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:03.608494 28766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:03.608736 28766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:03.611482 28766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "65736a494e9a4b769b02d14cde272d57"
format_stamp: "Formatted at 2026-08-09 15:01:03 on dist-test-slave-q9h9"
server_key: "f1f8722b2b0c7676bd1c3eb3f61d7c8b"
server_key_iv: "f68f48ec9e47066387d3d4c1c0ca3415"
server_key_version: "encryptionkey@0"
I20260809 15:01:03.612205 28766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "65736a494e9a4b769b02d14cde272d57"
format_stamp: "Formatted at 2026-08-09 15:01:03 on dist-test-slave-q9h9"
server_key: "f1f8722b2b0c7676bd1c3eb3f61d7c8b"
server_key_iv: "f68f48ec9e47066387d3d4c1c0ca3415"
server_key_version: "encryptionkey@0"
I20260809 15:01:03.617084 28766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 15:01:03.620326 28789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:03.621490 28766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 15:01:03.621639 28766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "65736a494e9a4b769b02d14cde272d57"
format_stamp: "Formatted at 2026-08-09 15:01:03 on dist-test-slave-q9h9"
server_key: "f1f8722b2b0c7676bd1c3eb3f61d7c8b"
server_key_iv: "f68f48ec9e47066387d3d4c1c0ca3415"
server_key_version: "encryptionkey@0"
I20260809 15:01:03.621809 28766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:03.640041 28766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:03.643873 28766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:03.644124 28766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:03.645043 28766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:03.646363 28766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:03.646441 28766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:03.646553 28766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:03.646611 28766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:03.679405 28766 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:34965
I20260809 15:01:03.679461 28976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:34965 every 8 connection(s)
I20260809 15:01:03.680533 28766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:03.688537 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28766
I20260809 15:01:03.688728 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:03.689235 28394 external_mini_cluster.cc:1428] Setting key dbd2580101265c5c97361499dc3756a1
Aug 09 15:01:03 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287663, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:03.699273 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:03.682864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56243 (local address 127.27.186.190:42175)
0809 15:01:03.683062 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:03.683068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:03.684136 (+ 1068us) server_negotiation.cc:407] Connection header received
0809 15:01:03.685790 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:03.685795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:03.685859 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:03.685943 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:03.687302 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.688028 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.690139 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.690661 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.690788 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:03.694155 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:03.694160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:03.694163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:03.694196 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:03.696522 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:03.697174 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:03.697179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:03.697181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:03.697312 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:03.697712 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:03.697715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:03.697717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:03.697957 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:03.698126 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:03.698725 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 15:01:03.698941 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:01:03.700022 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:03.683342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42175 (local address 127.27.186.130:56243)
0809 15:01:03.683988 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:03.684023 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:03.685568 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:03.686157 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:03.686165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:03.686617 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:03.687126 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:03.687142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.688148 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.688152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:03.689957 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:03.689970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:03.690887 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:03.690895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:03.691175 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:03.691768 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:03.691789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:03.693917 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:03.696654 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:03.696663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:03.696676 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:03.697048 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:03.697413 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:03.697417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:03.697421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:03.697593 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:03.698093 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:03.698100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:03.698386 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 15:01:03.698600 (+ 214us) client_negotiation.cc:240] Negotiation successful
0809 15:01:03.699039 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
I20260809 15:01:03.701224 28980 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42175
I20260809 15:01:03.701457 28980 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:03.702121 28980 heartbeater.cc:511] Master 127.27.186.190:42175 requested a full tablet report, sending...
I20260809 15:01:03.703428 28469 ts_manager.cc:194] Registered new tserver with Master: 65736a494e9a4b769b02d14cde272d57 (127.27.186.130:34965)
I20260809 15:01:03.704470 28469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:56243
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:03.726979 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42175
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:04.033519 28986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:04.033774 28986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:04.033914 28986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:04.041023 28986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:04.041124 28986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:04.041246 28986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:04.050374 28986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:04.051441 28986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:04.052628 28986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:04.064723 28994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:04.064690 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:04.064687 28995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:04.065431 28986 server_base.cc:1034] running on GCE node
I20260809 15:01:04.065923 28986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:04.066759 28986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:04.067948 28986 hybrid_clock.cc:648] HybridClock initialized: now 1786287664067896 us; error 48 us; skew 500 ppm
I20260809 15:01:04.068244 28986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:04.071108 28986 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:04.072507 28986 webserver.cc:460] Webserver started at http://127.27.186.131:33717/ using document root <none> and password file <none>
I20260809 15:01:04.073107 28986 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:04.073235 28986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:04.073578 28986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:04.076347 28986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "615d30ffd83c449fbd0805506026fefc"
format_stamp: "Formatted at 2026-08-09 15:01:04 on dist-test-slave-q9h9"
server_key: "762323e7ec77682cc473f8cf29548e3e"
server_key_iv: "d8856470ef3821477e0c84d91116c038"
server_key_version: "encryptionkey@0"
I20260809 15:01:04.077049 28986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "615d30ffd83c449fbd0805506026fefc"
format_stamp: "Formatted at 2026-08-09 15:01:04 on dist-test-slave-q9h9"
server_key: "762323e7ec77682cc473f8cf29548e3e"
server_key_iv: "d8856470ef3821477e0c84d91116c038"
server_key_version: "encryptionkey@0"
I20260809 15:01:04.081749 28986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260809 15:01:04.084951 29005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:04.086236 28986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 15:01:04.086372 28986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "615d30ffd83c449fbd0805506026fefc"
format_stamp: "Formatted at 2026-08-09 15:01:04 on dist-test-slave-q9h9"
server_key: "762323e7ec77682cc473f8cf29548e3e"
server_key_iv: "d8856470ef3821477e0c84d91116c038"
server_key_version: "encryptionkey@0"
I20260809 15:01:04.086541 28986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:04.098730 28986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:04.102347 28986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:04.102593 28986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:04.103456 28986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:04.104671 28986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:04.104749 28986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:04.104866 28986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:04.104925 28986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:04.139469 28986 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:41073
I20260809 15:01:04.139500 29175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:41073 every 8 connection(s)
I20260809 15:01:04.140813 28986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:04.150627 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28986
I20260809 15:01:04.150815 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:04.151223 28394 external_mini_cluster.cc:1428] Setting key 5c0909cdc65d4206ee59d2e5037ea414
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:04.160295 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.143502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:35201 (local address 127.27.186.190:42175)
0809 15:01:04.143670 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:04.143674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:04.144965 (+ 1291us) server_negotiation.cc:407] Connection header received
0809 15:01:04.146871 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:04.146877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:04.146990 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:04.147110 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:04.148640 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.149465 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.151085 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.151524 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.151636 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.155469 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:04.155474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:04.155477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:04.155508 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:04.157804 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.158492 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.158496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.158498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.158621 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.158978 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.158982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.158984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.159249 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:04.159357 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:04.159898 (+ 541us) server_negotiation.cc:299] Negotiation successful
0809 15:01:04.160047 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:01:04.161154 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.143995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42175 (local address 127.27.186.131:35201)
0809 15:01:04.144708 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:04.144748 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:04.146615 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:04.147344 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:04.147351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:04.147899 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:04.148462 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:04.148479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.149616 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.149620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:04.150958 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:04.150972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.151731 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.151739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:04.152001 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.152705 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:04.152738 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:04.155290 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:04.157943 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:04.157949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:04.157959 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:04.158395 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:04.158701 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:04.158704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:04.158706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:04.158887 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:04.159337 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:04.159342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:04.159614 (+ 272us) client_negotiation.cc:769] Sending connection context
0809 15:01:04.159813 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 15:01:04.160152 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
I20260809 15:01:04.162447 29180 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42175
I20260809 15:01:04.162691 29180 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:04.163371 29180 heartbeater.cc:511] Master 127.27.186.190:42175 requested a full tablet report, sending...
I20260809 15:01:04.164700 28469 ts_manager.cc:194] Registered new tserver with Master: 615d30ffd83c449fbd0805506026fefc (127.27.186.131:41073)
I20260809 15:01:04.165551 28469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:35201
I20260809 15:01:04.167006 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:04.214303 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.199454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.27.186.129:33511)
0809 15:01:04.200029 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:04.200036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:04.200236 (+ 200us) server_negotiation.cc:407] Connection header received
0809 15:01:04.200437 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:04.200444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:04.200649 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:04.201014 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:04.202282 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.203319 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.204478 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.205011 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.205110 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.208585 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:04.208595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:04.208608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:04.208639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:04.211610 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.212222 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.212229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.212231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.212387 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.212783 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.212788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.212792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.213105 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:04.213414 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:04.213696 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 15:01:04.213959 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
W20260809 15:01:04.215582 28627 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59882
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260809 15:01:04.240387 28757 heartbeater.cc:503] Master 127.27.186.190:42175 was elected leader, sending a full tablet report...
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:04.260910 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.246329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.27.186.129:33511)
0809 15:01:04.246525 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:04.246532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:04.246817 (+ 285us) server_negotiation.cc:407] Connection header received
0809 15:01:04.246983 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:04.246986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:04.247077 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:04.247178 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:04.248431 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.249115 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.250408 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.251005 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.251133 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.255394 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:04.255399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:04.255403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:04.255434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:04.258270 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.258920 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.258925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.258928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.259067 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.259455 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.259460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.259463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.259746 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:04.259993 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:04.260327 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 15:01:04.260561 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 15:01:04.262624 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28561
I20260809 15:01:04.274358 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.263549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.27.186.129:33511)
0809 15:01:04.263730 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:04.263736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:04.264010 (+ 274us) server_negotiation.cc:407] Connection header received
0809 15:01:04.264141 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:04.264145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:04.264203 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:04.264279 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:04.265415 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.266116 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.267311 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.267728 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.267835 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.269814 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:04.269818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:04.269821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:04.269849 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:04.272036 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.272608 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.272612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.272614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.272760 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.273088 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.273092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.273094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.273336 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:04.273554 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:04.273830 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 15:01:04.274048 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:01:04.274962 28746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:04.355996 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28561
I20260809 15:01:04.380816 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28766
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:04.396556 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.381725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.27.186.130:34965)
0809 15:01:04.382273 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:04.382280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:04.382308 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:01:04.382381 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:04.382387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:04.382603 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:04.382746 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:04.383847 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.384913 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.386266 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.386840 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.386951 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.390906 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:04.390922 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:04.390937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:04.390976 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:04.394089 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.394695 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.394702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.394704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.394854 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.395150 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.395153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.395155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.395499 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:04.395755 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:04.395971 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 15:01:04.396177 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260809 15:01:04.397800 28970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:04.476876 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28766
I20260809 15:01:04.495790 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28986
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:04.510476 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:04.496647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.27.186.131:41073)
0809 15:01:04.497229 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:04.497237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:04.497264 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:01:04.497321 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:04.497327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:04.497522 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:04.497663 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:04.498939 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.499950 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.501251 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:04.501785 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:04.501900 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:04.505676 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:04.505687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:04.505700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:04.505734 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:04.508175 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.508758 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.508766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.508770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.508919 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:04.509198 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:04.509201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:04.509203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:04.509534 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:04.509759 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:04.509964 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 15:01:04.510190 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":278,"threads_started":1}
I20260809 15:01:04.511713 29171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:04.592129 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28986
I20260809 15:01:04.614856 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28422
I20260809 15:01:04.615926 28516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:04.715147 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28422
2026-08-09T15:01:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4608 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[29209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[29209](info): set up 2 sockets
Aug 09 15:01:04 dist-test-slave-q9h9 krb5kdc[29209](info): commencing operation
krb5kdc: starting...
W20260809 15:01:05.889169 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:05 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:06.043270 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42631
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:42631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:06.344590 29228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:06.344830 29228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:06.344940 29228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:06.351714 29228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:06.351802 29228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:06.351873 29228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:06.351945 29228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:06.352003 29228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:06.361044 29228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:42631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:06.362095 29228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:06.363230 29228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:06.374140 29244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:06.374225 29237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:06.374199 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:06.374517 29228 server_base.cc:1034] running on GCE node
I20260809 15:01:06.375449 29228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:06.376643 29228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:06.377836 29228 hybrid_clock.cc:648] HybridClock initialized: now 1786287666377787 us; error 35 us; skew 500 ppm
I20260809 15:01:06.378165 29228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:06 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287666, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:06.386313 29228 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:06.387737 29228 webserver.cc:460] Webserver started at http://127.27.186.190:44765/ using document root <none> and password file <none>
I20260809 15:01:06.388343 29228 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:06.388459 29228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:06.388747 29228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:06.391559 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "e283c6fa9eff44fcb3cbb446d2346d00"
format_stamp: "Formatted at 2026-08-09 15:01:06 on dist-test-slave-q9h9"
server_key: "d9d64bfeb1ce0b413a08e7e453a54faf"
server_key_iv: "f48081723152a2093dcb4911ce86b4ca"
server_key_version: "encryptionkey@0"
I20260809 15:01:06.392251 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "e283c6fa9eff44fcb3cbb446d2346d00"
format_stamp: "Formatted at 2026-08-09 15:01:06 on dist-test-slave-q9h9"
server_key: "d9d64bfeb1ce0b413a08e7e453a54faf"
server_key_iv: "f48081723152a2093dcb4911ce86b4ca"
server_key_version: "encryptionkey@0"
I20260809 15:01:06.396986 29228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 15:01:06.400151 29254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:06.401283 29228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:06.401438 29228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "e283c6fa9eff44fcb3cbb446d2346d00"
format_stamp: "Formatted at 2026-08-09 15:01:06 on dist-test-slave-q9h9"
server_key: "d9d64bfeb1ce0b413a08e7e453a54faf"
server_key_iv: "f48081723152a2093dcb4911ce86b4ca"
server_key_version: "encryptionkey@0"
I20260809 15:01:06.401607 29228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:06.418977 29228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:06.422412 29228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:06.422641 29228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:06.439045 29228 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:42631
I20260809 15:01:06.439067 29337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:42631 every 8 connection(s)
I20260809 15:01:06.440196 29228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:06.443477 29338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:06.445868 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29228
I20260809 15:01:06.446091 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:06.446722 28394 external_mini_cluster.cc:1428] Setting key f3fc61d49be4216b1022cdce798f6585
I20260809 15:01:06.454698 29338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00: Bootstrap starting.
I20260809 15:01:06.457515 29338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:06.458453 29338 log.cc:824] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:01:06 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:06.460827 29338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00: No bootstrap required, opened a new log
I20260809 15:01:06.467053 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:06.448501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.27.186.190:42631)
0809 15:01:06.449277 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:06.449288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:06.449325 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:01:06.450903 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:06.450925 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:06.451276 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:06.451577 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:06.453420 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:06.454710 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:06.455919 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:06.456538 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:06.456690 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:06.460377 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:06.460384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:06.460396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:06.460433 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:06.463341 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:06.463977 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:06.463983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:06.463989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:06.464157 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:06.464511 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:06.464514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:06.464516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:06.465020 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:06.465305 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:06.465789 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 15:01:06.466175 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":326,"threads_started":1}
I20260809 15:01:06.468245 29338 raft_consensus.cc:348] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42631 } }
I20260809 15:01:06.468567 29338 raft_consensus.cc:374] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:06.468660 29338 raft_consensus.cc:729] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e283c6fa9eff44fcb3cbb446d2346d00, State: Initialized, Role: FOLLOWER
I20260809 15:01:06.469146 29338 consensus_queue.cc:260] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42631 } }
I20260809 15:01:06.469383 29338 raft_consensus.cc:388] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:06.469522 29338 raft_consensus.cc:482] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:06.469672 29338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:06.471624 29338 raft_consensus.cc:504] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42631 } }
I20260809 15:01:06.472047 29338 leader_election.cc:302] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e283c6fa9eff44fcb3cbb446d2346d00; no voters:
I20260809 15:01:06.472563 29338 leader_election.cc:288] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:06.472666 29346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:06.473769 29346 raft_consensus.cc:686] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [term 1 LEADER]: Becoming Leader. State: Replica: e283c6fa9eff44fcb3cbb446d2346d00, State: Running, Role: LEADER
I20260809 15:01:06.474347 29346 consensus_queue.cc:237] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42631 } }
I20260809 15:01:06.474632 29338 sys_catalog.cc:564] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:06.477264 29349 sys_catalog.cc:455] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e283c6fa9eff44fcb3cbb446d2346d00. Latest consensus state: current_term: 1 leader_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42631 } } }
I20260809 15:01:06.477118 29347 sys_catalog.cc:455] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e283c6fa9eff44fcb3cbb446d2346d00" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42631 } } }
I20260809 15:01:06.477595 29349 sys_catalog.cc:458] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:06.477661 29347 sys_catalog.cc:458] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:06.478549 29360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:06.483132 29360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:06.490981 29360 catalog_manager.cc:1347] Generated new cluster ID: 106055a63bda48f2ac623a8c700235a6
I20260809 15:01:06.491081 29360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:06.509109 29360 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:06.510685 29360 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:06.517695 29360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e283c6fa9eff44fcb3cbb446d2346d00: Generated new TSK 0
I20260809 15:01:06.518425 29360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:06.563342 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42631
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:06.887681 29381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:06.887914 29381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:06.888012 29381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:06.894840 29381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:06.894927 29381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:06.895038 29381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:06.904009 29381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:06.905066 29381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:06.906324 29381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:06.918118 29394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:06.918265 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:06.918161 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:06.919059 29381 server_base.cc:1034] running on GCE node
I20260809 15:01:06.919579 29381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:06.920369 29381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:06.921556 29381 hybrid_clock.cc:648] HybridClock initialized: now 1786287666921504 us; error 42 us; skew 500 ppm
I20260809 15:01:06.921828 29381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:06 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287666, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:06.924736 29381 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:06.926096 29381 webserver.cc:460] Webserver started at http://127.27.186.129:38311/ using document root <none> and password file <none>
I20260809 15:01:06.926695 29381 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:06.926815 29381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:06.927106 29381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:06.929672 29381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "76310913a225423bb4279ffce36617e9"
format_stamp: "Formatted at 2026-08-09 15:01:06 on dist-test-slave-q9h9"
server_key: "dee8407c1a36c07668b7cc1f662abd2f"
server_key_iv: "d45dc74c42376c15872168d4eb5894db"
server_key_version: "encryptionkey@0"
I20260809 15:01:06.930416 29381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "76310913a225423bb4279ffce36617e9"
format_stamp: "Formatted at 2026-08-09 15:01:06 on dist-test-slave-q9h9"
server_key: "dee8407c1a36c07668b7cc1f662abd2f"
server_key_iv: "d45dc74c42376c15872168d4eb5894db"
server_key_version: "encryptionkey@0"
I20260809 15:01:06.934950 29381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:01:06.938050 29404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:06.939265 29381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:01:06.939397 29381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "76310913a225423bb4279ffce36617e9"
format_stamp: "Formatted at 2026-08-09 15:01:06 on dist-test-slave-q9h9"
server_key: "dee8407c1a36c07668b7cc1f662abd2f"
server_key_iv: "d45dc74c42376c15872168d4eb5894db"
server_key_version: "encryptionkey@0"
I20260809 15:01:06.939561 29381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:06.975586 29381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:06.979578 29381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:06.979823 29381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:06.980684 29381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:06.981875 29381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:06.981963 29381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:06.982096 29381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:06.982152 29381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.011742 29381 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:35079
I20260809 15:01:07.011809 29588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:35079 every 8 connection(s)
I20260809 15:01:07.012887 29381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:07.018970 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29381
I20260809 15:01:07.019164 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:07.019842 28394 external_mini_cluster.cc:1428] Setting key f4c26a56301cea5c429de6354c009705
Aug 09 15:01:07 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287666, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:07.032229 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.015183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:60261 (local address 127.27.186.190:42631)
0809 15:01:07.015627 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:07.015633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:07.016501 (+ 868us) server_negotiation.cc:407] Connection header received
0809 15:01:07.018606 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:07.018612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:07.018699 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:07.018824 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:07.020247 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.021135 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.022847 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.023460 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.023583 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.026763 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:07.026768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:07.026773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:07.026814 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:07.029311 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.030163 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.030168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.030170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.030312 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.030743 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.030746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.030748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.030991 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:07.031103 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:07.031818 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 15:01:07.031979 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":211,"threads_started":1}
I20260809 15:01:07.033416 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.015688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42631 (local address 127.27.186.129:60261)
0809 15:01:07.016356 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:07.016392 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:07.018355 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:07.018976 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:07.018983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:07.019536 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:07.020077 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:07.020100 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.021283 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.021288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:07.022689 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:07.022703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.023636 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.023640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:07.023831 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.024420 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:07.024441 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:07.026557 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:07.029442 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:07.029450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:07.029464 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:07.029995 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:07.030417 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:07.030421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:07.030425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:07.030647 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:07.031114 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:07.031121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:07.031481 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 15:01:07.031739 (+ 258us) client_negotiation.cc:240] Negotiation successful
0809 15:01:07.032169 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260809 15:01:07.034860 29591 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42631
I20260809 15:01:07.035126 29591 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:07.035774 29591 heartbeater.cc:511] Master 127.27.186.190:42631 requested a full tablet report, sending...
I20260809 15:01:07.037926 29284 ts_manager.cc:194] Registered new tserver with Master: 76310913a225423bb4279ffce36617e9 (127.27.186.129:35079)
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
I20260809 15:01:07.039616 29284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:60261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:07.058252 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42631
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:07.368330 29604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:07.368584 29604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:07.368726 29604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:07.375784 29604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:07.375892 29604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:07.376034 29604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:07.385284 29604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:07.386413 29604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:07.387598 29604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:07.399672 29610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:07.399724 29611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:07.399750 29616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:07.400538 29604 server_base.cc:1034] running on GCE node
I20260809 15:01:07.401054 29604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:07.401877 29604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:07.403090 29604 hybrid_clock.cc:648] HybridClock initialized: now 1786287667403027 us; error 51 us; skew 500 ppm
I20260809 15:01:07.403388 29604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:07 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287667, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:07.406368 29604 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:07.407667 29604 webserver.cc:460] Webserver started at http://127.27.186.130:39367/ using document root <none> and password file <none>
I20260809 15:01:07.408272 29604 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:07.408377 29604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:07.408666 29604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:07.411388 29604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "f3096b71e4ed4bbc80ad487d0ad516a0"
format_stamp: "Formatted at 2026-08-09 15:01:07 on dist-test-slave-q9h9"
server_key: "d706c25cbc92cf6568895261481f7b86"
server_key_iv: "187541a49ae09c129717960f63be23e3"
server_key_version: "encryptionkey@0"
I20260809 15:01:07.412120 29604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "f3096b71e4ed4bbc80ad487d0ad516a0"
format_stamp: "Formatted at 2026-08-09 15:01:07 on dist-test-slave-q9h9"
server_key: "d706c25cbc92cf6568895261481f7b86"
server_key_iv: "187541a49ae09c129717960f63be23e3"
server_key_version: "encryptionkey@0"
I20260809 15:01:07.417224 29604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 15:01:07.420452 29627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.421605 29604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:07.421763 29604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "f3096b71e4ed4bbc80ad487d0ad516a0"
format_stamp: "Formatted at 2026-08-09 15:01:07 on dist-test-slave-q9h9"
server_key: "d706c25cbc92cf6568895261481f7b86"
server_key_iv: "187541a49ae09c129717960f63be23e3"
server_key_version: "encryptionkey@0"
I20260809 15:01:07.421940 29604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:07.439625 29604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:07.445144 29604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:07.445410 29604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:07.446527 29604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:07.448019 29604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:07.448102 29604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.448210 29604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:07.448264 29604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.478379 29604 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33863
I20260809 15:01:07.478426 29813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33863 every 8 connection(s)
I20260809 15:01:07.479453 29604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:07.481320 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29604
I20260809 15:01:07.481493 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:07.482048 28394 external_mini_cluster.cc:1428] Setting key fd2ce87696b8e54f42a3784b623551ac
Aug 09 15:01:07 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287667, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:07.499842 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.481868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:52051 (local address 127.27.186.190:42631)
0809 15:01:07.482088 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:07.482094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:07.483210 (+ 1116us) server_negotiation.cc:407] Connection header received
0809 15:01:07.485429 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:07.485432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:07.485494 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:07.485573 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:07.487432 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.488183 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.490119 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.490543 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.490680 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.494297 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:07.494302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:07.494306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:07.494338 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:07.496828 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.497655 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.497659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.497661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.497784 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.498295 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.498299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.498300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.498544 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:07.498663 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:07.499457 (+ 794us) server_negotiation.cc:299] Negotiation successful
0809 15:01:07.499614 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
I20260809 15:01:07.501093 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.482308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42631 (local address 127.27.186.130:52051)
0809 15:01:07.483064 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:07.483110 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:07.485159 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:07.485835 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:07.485846 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:07.486534 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:07.487250 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:07.487273 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.488342 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.488348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:07.489964 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:07.489979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.490746 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.490751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:07.490929 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.491550 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:07.491575 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:07.494037 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:07.496969 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:07.496978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:07.496991 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:07.497538 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:07.497887 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:07.497892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:07.497896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:07.498186 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:07.498679 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:07.498686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:07.499081 (+ 395us) client_negotiation.cc:769] Sending connection context
0809 15:01:07.499379 (+ 298us) client_negotiation.cc:240] Negotiation successful
0809 15:01:07.499871 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":307,"threads_started":1}
I20260809 15:01:07.502383 29814 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42631
I20260809 15:01:07.502643 29814 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
I20260809 15:01:07.503333 29814 heartbeater.cc:511] Master 127.27.186.190:42631 requested a full tablet report, sending...
I20260809 15:01:07.504767 29284 ts_manager.cc:194] Registered new tserver with Master: f3096b71e4ed4bbc80ad487d0ad516a0 (127.27.186.130:33863)
I20260809 15:01:07.505589 29284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:52051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:07.520546 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42631
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:07.825037 29821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:07.825291 29821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:07.825440 29821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:07.832490 29821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:07.832608 29821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:07.832762 29821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:07.841988 29821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:07.843133 29821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:07.844393 29821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:07.856351 29827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:07.856382 29840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:07.856407 29831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:07.857456 29821 server_base.cc:1034] running on GCE node
I20260809 15:01:07.858058 29821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:07.858882 29821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:07.860069 29821 hybrid_clock.cc:648] HybridClock initialized: now 1786287667860019 us; error 43 us; skew 500 ppm
I20260809 15:01:07.860344 29821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:07 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287667, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:07.863255 29821 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:07.864648 29821 webserver.cc:460] Webserver started at http://127.27.186.131:36545/ using document root <none> and password file <none>
I20260809 15:01:07.865239 29821 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:07.865357 29821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:07.865649 29821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:07.868304 29821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "7e21020481e040cd8c7556282f337be8"
format_stamp: "Formatted at 2026-08-09 15:01:07 on dist-test-slave-q9h9"
server_key: "1d263c205e716c878f48b23be0f9614c"
server_key_iv: "9c1ee72028918c31971d366d7b5b3634"
server_key_version: "encryptionkey@0"
I20260809 15:01:07.869032 29821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "7e21020481e040cd8c7556282f337be8"
format_stamp: "Formatted at 2026-08-09 15:01:07 on dist-test-slave-q9h9"
server_key: "1d263c205e716c878f48b23be0f9614c"
server_key_iv: "9c1ee72028918c31971d366d7b5b3634"
server_key_version: "encryptionkey@0"
I20260809 15:01:07.873692 29821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:01:07.877141 29854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.878437 29821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:07.878578 29821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "7e21020481e040cd8c7556282f337be8"
format_stamp: "Formatted at 2026-08-09 15:01:07 on dist-test-slave-q9h9"
server_key: "1d263c205e716c878f48b23be0f9614c"
server_key_iv: "9c1ee72028918c31971d366d7b5b3634"
server_key_version: "encryptionkey@0"
I20260809 15:01:07.878765 29821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:07.908399 29821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:07.911906 29821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:07.912153 29821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:07.913018 29821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:07.914237 29821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:07.914314 29821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.914430 29821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:07.914494 29821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:07.945602 29821 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:36125
I20260809 15:01:07.945644 30039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:36125 every 8 connection(s)
I20260809 15:01:07.946730 29821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:07.953047 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29821
I20260809 15:01:07.953259 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:07.953783 28394 external_mini_cluster.cc:1428] Setting key 370c160a745b46ada5629811cad34b66
Aug 09 15:01:07 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287667, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:07.965009 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.948809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37261 (local address 127.27.186.190:42631)
0809 15:01:07.949000 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:07.949007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:07.950125 (+ 1118us) server_negotiation.cc:407] Connection header received
0809 15:01:07.951893 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:07.951897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:07.951963 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:07.952048 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:07.953732 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.954545 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.956381 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.956857 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.956954 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.959990 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:07.959994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:07.959998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:07.960031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:07.962276 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.962911 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.962916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.962918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.963044 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.963441 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.963446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.963449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.963718 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:07.963862 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:07.964446 (+ 584us) server_negotiation.cc:299] Negotiation successful
0809 15:01:07.964669 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:01:07.965763 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.949290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42631 (local address 127.27.186.131:37261)
0809 15:01:07.949953 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:07.949993 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:07.951637 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:07.952269 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:07.952276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:07.952832 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:07.953559 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:07.953584 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.954697 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.954701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:07.956249 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:07.956266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.957038 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.957043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:07.957232 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.957760 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:07.957779 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:07.959802 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:07.962405 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:07.962414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:07.962427 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:07.962801 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:07.963153 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:07.963158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:07.963161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:07.963335 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:07.963839 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:07.963846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:07.964117 (+ 271us) client_negotiation.cc:769] Sending connection context
0809 15:01:07.964350 (+ 233us) client_negotiation.cc:240] Negotiation successful
0809 15:01:07.964777 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260809 15:01:07.966897 30040 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42631
I20260809 15:01:07.967180 30040 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:07.968104 30040 heartbeater.cc:511] Master 127.27.186.190:42631 requested a full tablet report, sending...
I20260809 15:01:07.969439 29284 ts_manager.cc:194] Registered new tserver with Master: 7e21020481e040cd8c7556282f337be8 (127.27.186.131:36125)
I20260809 15:01:07.970237 29284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:37261
I20260809 15:01:07.980260 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:01:07.997169 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:07.986281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.27.186.190:42631)
0809 15:01:07.986458 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:07.986464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:07.986944 (+ 480us) server_negotiation.cc:407] Connection header received
0809 15:01:07.987052 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:07.987055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:07.987108 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:07.987180 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:07.988513 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.989186 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.990419 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:07.990793 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:07.990880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:07.992862 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:07.992866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:07.992869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:07.992891 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:07.995147 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.995706 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.995710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.995712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.995834 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:07.996088 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:07.996091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:07.996093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:07.996370 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:07.996532 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:07.996753 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 15:01:07.996897 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:01:08.002490 29284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:08.009291 29284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:08.042140 29591 heartbeater.cc:503] Master 127.27.186.190:42631 was elected leader, sending a full tablet report...
I20260809 15:01:08.043834 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.021088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:35079 (local address 127.0.0.1:47364)
0809 15:01:08.033918 (+ 12830us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:08.033965 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:08.034729 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:08.035572 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:08.035580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:08.035633 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:08.036102 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.036115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.038321 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.038326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.040708 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.040727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.042899 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.042907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.043040 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.043071 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:01:08.043206 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 15:01:08.043378 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12580,"mutex_wait_us":57,"spinlock_wait_cycles":7680,"thread_start_us":12933,"threads_started":1}
I20260809 15:01:08.043836 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.034994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.27.186.130:33863)
0809 15:01:08.035516 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.035524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.036256 (+ 732us) server_negotiation.cc:407] Connection header received
0809 15:01:08.036412 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.036419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.036576 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.036778 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:08.037356 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.038697 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.041162 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.042418 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.042575 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.042749 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.043343 (+ 594us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.043505 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
I20260809 15:01:08.044377 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.021172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:36125 (local address 127.0.0.1:45026)
0809 15:01:08.034281 (+ 13109us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:08.034299 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:08.034446 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:08.035573 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:08.035580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:08.035631 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:08.036507 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.036520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.038218 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.038223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.040287 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.040299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.042677 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.042683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.043680 (+ 997us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.043721 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 15:01:08.043826 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 15:01:08.043943 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12964,"mutex_wait_us":435,"spinlock_wait_cycles":2176,"thread_start_us":12854,"threads_started":1}
I20260809 15:01:08.044447 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.021235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.27.186.129:35079)
0809 15:01:08.033594 (+ 12359us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.033604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.034635 (+ 1031us) server_negotiation.cc:407] Connection header received
0809 15:01:08.034935 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.034943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.035146 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.035348 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:08.036307 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.038208 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.040868 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.042314 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.043295 (+ 981us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.043444 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.043671 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.043841 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12189,"thread_start_us":12471,"threads_started":1}
I20260809 15:01:08.044692 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.035560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33863 (local address 127.0.0.1:53058)
0809 15:01:08.036134 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:08.036150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:08.036271 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:08.036765 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:08.036769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:08.036793 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:08.037159 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.037172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.038796 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.038801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.041021 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.041031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.042950 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.042956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.043111 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.043140 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 15:01:08.044126 (+ 986us) client_negotiation.cc:240] Negotiation successful
0809 15:01:08.044263 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":32,"thread_start_us":310,"threads_started":1}
I20260809 15:01:08.044872 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.021603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.27.186.131:36125)
0809 15:01:08.035030 (+ 13427us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.035037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.035066 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:01:08.035144 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.035149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.035351 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.035975 (+ 624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:08.036694 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.038110 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.040542 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.041842 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.042696 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.042858 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.044422 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.044580 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13272,"thread_start_us":13106,"threads_started":1}
I20260809 15:01:08.047847 29934 tablet_service.cc:1467] Processing CreateTablet for tablet 7e85560d73f94c5988de84def513c9f4 (DEFAULT_TABLE table=test-table [id=69070b442219431b964778dbe49fb8f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:08.048086 29704 tablet_service.cc:1467] Processing CreateTablet for tablet 7e85560d73f94c5988de84def513c9f4 (DEFAULT_TABLE table=test-table [id=69070b442219431b964778dbe49fb8f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:08.048091 29477 tablet_service.cc:1467] Processing CreateTablet for tablet 7e85560d73f94c5988de84def513c9f4 (DEFAULT_TABLE table=test-table [id=69070b442219431b964778dbe49fb8f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:08.049199 29934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e85560d73f94c5988de84def513c9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:08.049360 29477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e85560d73f94c5988de84def513c9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:08.049383 29704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e85560d73f94c5988de84def513c9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:08.059916 30076 tablet_bootstrap.cc:492] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: Bootstrap starting.
I20260809 15:01:08.063053 30078 tablet_bootstrap.cc:492] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: Bootstrap starting.
I20260809 15:01:08.063943 30076 tablet_bootstrap.cc:654] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:08.065213 30076 log.cc:824] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:08.066599 30079 tablet_bootstrap.cc:492] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: Bootstrap starting.
I20260809 15:01:08.066653 30078 tablet_bootstrap.cc:654] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:08.067862 30078 log.cc:824] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:08.068081 30076 tablet_bootstrap.cc:492] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: No bootstrap required, opened a new log
I20260809 15:01:08.068322 30076 ts_tablet_manager.cc:1397] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260809 15:01:08.069793 30079 tablet_bootstrap.cc:654] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:08.070945 30078 tablet_bootstrap.cc:492] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: No bootstrap required, opened a new log
I20260809 15:01:08.070998 30079 log.cc:824] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:08.071220 30078 ts_tablet_manager.cc:1397] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:01:08.074219 30079 tablet_bootstrap.cc:492] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: No bootstrap required, opened a new log
I20260809 15:01:08.074479 30079 ts_tablet_manager.cc:1397] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260809 15:01:08.076490 30076 raft_consensus.cc:348] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.076819 30076 raft_consensus.cc:374] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:08.076928 30076 raft_consensus.cc:729] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e21020481e040cd8c7556282f337be8, State: Initialized, Role: FOLLOWER
I20260809 15:01:08.077509 30076 consensus_queue.cc:260] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.078492 30040 heartbeater.cc:503] Master 127.27.186.190:42631 was elected leader, sending a full tablet report...
I20260809 15:01:08.080103 30076 ts_tablet_manager.cc:1428] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260809 15:01:08.080869 30078 raft_consensus.cc:348] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.081278 30078 raft_consensus.cc:374] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:08.081382 30078 raft_consensus.cc:729] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3096b71e4ed4bbc80ad487d0ad516a0, State: Initialized, Role: FOLLOWER
I20260809 15:01:08.082042 30078 consensus_queue.cc:260] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.083050 29814 heartbeater.cc:503] Master 127.27.186.190:42631 was elected leader, sending a full tablet report...
I20260809 15:01:08.084342 30078 ts_tablet_manager.cc:1428] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260809 15:01:08.085619 30079 raft_consensus.cc:348] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.086053 30079 raft_consensus.cc:374] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:08.086155 30079 raft_consensus.cc:729] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76310913a225423bb4279ffce36617e9, State: Initialized, Role: FOLLOWER
I20260809 15:01:08.086673 30079 consensus_queue.cc:260] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.088042 30079 ts_tablet_manager.cc:1428] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260809 15:01:08.198271 30041 tablet.cc:2369] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:08.202193 30087 raft_consensus.cc:482] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:08.202436 30087 raft_consensus.cc:504] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.204478 30087 leader_election.cc:288] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e21020481e040cd8c7556282f337be8 (127.27.186.131:36125), f3096b71e4ed4bbc80ad487d0ad516a0 (127.27.186.130:33863)
I20260809 15:01:08.210757 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.204965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:55151 (local address 127.27.186.131:36125)
0809 15:01:08.205155 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.205161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.205615 (+ 454us) server_negotiation.cc:407] Connection header received
0809 15:01:08.205859 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.205866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.205954 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.206069 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:08.206700 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.207404 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.209099 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.209863 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.210005 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.210106 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.210378 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.210499 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:01:08.210872 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.204853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:36125 (local address 127.27.186.129:55151)
0809 15:01:08.205515 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:08.205530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:08.205671 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:08.206194 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:08.206199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:08.206243 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:08.206556 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.206565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.207526 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.207532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.208950 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.208963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.210087 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.210093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.210234 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.210257 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:01:08.210340 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 15:01:08.210457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"spinlock_wait_cycles":6016,"thread_start_us":333,"threads_started":1}
I20260809 15:01:08.211331 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.205967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:35735 (local address 127.27.186.130:33863)
0809 15:01:08.206201 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.206207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.206323 (+ 116us) server_negotiation.cc:407] Connection header received
0809 15:01:08.206529 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.206533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.206591 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.206664 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:08.207177 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.207885 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.209569 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.210360 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.210492 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.210558 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.210960 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.211081 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 15:01:08.211933 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.205662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33863 (local address 127.27.186.129:35735)
0809 15:01:08.206228 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:08.206250 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:08.206401 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:08.206694 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:08.206698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:08.206721 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:08.207041 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.207049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.208017 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.208022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.209423 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:08.209435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.210563 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.210570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:08.210757 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.210790 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:01:08.211488 (+ 698us) client_negotiation.cc:240] Negotiation successful
0809 15:01:08.211616 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":258,"threads_started":1}
I20260809 15:01:08.212026 29966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e85560d73f94c5988de84def513c9f4" candidate_uuid: "76310913a225423bb4279ffce36617e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e21020481e040cd8c7556282f337be8" is_pre_election: true
I20260809 15:01:08.212827 29966 raft_consensus.cc:2454] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76310913a225423bb4279ffce36617e9 in term 0.
I20260809 15:01:08.212950 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e85560d73f94c5988de84def513c9f4" candidate_uuid: "76310913a225423bb4279ffce36617e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" is_pre_election: true
I20260809 15:01:08.213490 29735 raft_consensus.cc:2454] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76310913a225423bb4279ffce36617e9 in term 0.
I20260809 15:01:08.213686 29405 leader_election.cc:302] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76310913a225423bb4279ffce36617e9, 7e21020481e040cd8c7556282f337be8; no voters:
I20260809 15:01:08.214102 30087 raft_consensus.cc:2781] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:08.214253 30087 raft_consensus.cc:482] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:08.214386 30087 raft_consensus.cc:3037] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:08.216511 30087 raft_consensus.cc:504] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.217061 30087 leader_election.cc:288] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [CANDIDATE]: Term 1 election: Requested vote from peers 7e21020481e040cd8c7556282f337be8 (127.27.186.131:36125), f3096b71e4ed4bbc80ad487d0ad516a0 (127.27.186.130:33863)
I20260809 15:01:08.217381 29966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e85560d73f94c5988de84def513c9f4" candidate_uuid: "76310913a225423bb4279ffce36617e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e21020481e040cd8c7556282f337be8"
I20260809 15:01:08.217478 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e85560d73f94c5988de84def513c9f4" candidate_uuid: "76310913a225423bb4279ffce36617e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0"
I20260809 15:01:08.217549 29966 raft_consensus.cc:3037] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:08.217653 29735 raft_consensus.cc:3037] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:08.219717 29966 raft_consensus.cc:2454] T 7e85560d73f94c5988de84def513c9f4 P 7e21020481e040cd8c7556282f337be8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76310913a225423bb4279ffce36617e9 in term 1.
I20260809 15:01:08.219794 29735 raft_consensus.cc:2454] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76310913a225423bb4279ffce36617e9 in term 1.
I20260809 15:01:08.220103 29405 leader_election.cc:302] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76310913a225423bb4279ffce36617e9, 7e21020481e040cd8c7556282f337be8; no voters:
I20260809 15:01:08.220361 30087 raft_consensus.cc:2781] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:08.220769 30087 raft_consensus.cc:686] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [term 1 LEADER]: Becoming Leader. State: Replica: 76310913a225423bb4279ffce36617e9, State: Running, Role: LEADER
I20260809 15:01:08.221215 30087 consensus_queue.cc:237] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } }
I20260809 15:01:08.226280 29284 catalog_manager.cc:5495] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76310913a225423bb4279ffce36617e9 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "76310913a225423bb4279ffce36617e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76310913a225423bb4279ffce36617e9" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e21020481e040cd8c7556282f337be8" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3096b71e4ed4bbc80ad487d0ad516a0" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33863 } health_report { overall_health: UNKNOWN } } }
W20260809 15:01:08.231309 29815 tablet.cc:2369] T 7e85560d73f94c5988de84def513c9f4 P f3096b71e4ed4bbc80ad487d0ad516a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:01:08.265105 29597 tablet.cc:2369] T 7e85560d73f94c5988de84def513c9f4 P 76310913a225423bb4279ffce36617e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:08.274206 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.259363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47380 (local address 127.27.186.129:35079)
0809 15:01:08.259587 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.259593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.260086 (+ 493us) server_negotiation.cc:407] Connection header received
0809 15:01:08.260212 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.260215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.260279 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.260362 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:08.261738 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.262500 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.263724 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.264249 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.264378 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.268824 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:08.268837 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:08.268851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:08.268889 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:08.271611 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.272236 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.272244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.272246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.272405 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.272749 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.272753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.272755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.273094 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:08.273382 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.273656 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.273879 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287668, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287668, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:08.324254 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.311435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.27.186.129:35079)
0809 15:01:08.311645 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.311653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.311951 (+ 298us) server_negotiation.cc:407] Connection header received
0809 15:01:08.312120 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.312125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.312296 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.312413 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:08.313754 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.314505 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.315795 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.316203 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.316312 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.319869 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:08.319874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:08.319878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:08.319910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:08.322204 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.322715 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.322718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.322720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.322825 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.323158 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.323161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.323163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.323408 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:08.323624 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.323769 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.323927 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:08.365067 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.352662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.27.186.129:35079)
0809 15:01:08.352860 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.352866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.353144 (+ 278us) server_negotiation.cc:407] Connection header received
0809 15:01:08.353264 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.353268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.353341 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.353417 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:08.354717 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.355472 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.356574 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.357010 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.357100 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.360691 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:08.360696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:08.360699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:08.360729 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:08.362974 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.363513 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.363516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.363518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.363633 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.363944 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.363947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.363949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.364211 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:08.364406 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.364602 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.364754 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 15:01:08.367311 29443 tablet_service.cc:2908] Rejecting scan request for tablet 7e85560d73f94c5988de84def513c9f4: Uninitialized: clean time has not yet been initialized
I20260809 15:01:08.370540 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29381
I20260809 15:01:08.381959 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.371331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.27.186.129:35079)
0809 15:01:08.371469 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.371473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.371782 (+ 309us) server_negotiation.cc:407] Connection header received
0809 15:01:08.371941 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.371944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.372008 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.372081 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:08.373120 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.373805 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.374953 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.375389 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.375483 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.377464 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:08.377469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:08.377472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:08.377499 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:08.379762 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.380416 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.380420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.380423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.380545 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.380816 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.380819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.380821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.381063 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:08.381263 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.381471 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.381651 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:01:08.382588 29579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:08.475733 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29381
I20260809 15:01:08.500013 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29604
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:08.513818 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.500675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.27.186.130:33863)
0809 15:01:08.500848 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.500853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.500872 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:08.501040 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.501044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.501104 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.501183 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:08.502385 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.503133 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.504386 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.504791 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.504897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.508731 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:08.508743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:08.508756 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:08.508787 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:08.511304 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.511920 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.511927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.511929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.512050 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.512397 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.512401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.512403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.512753 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:08.513053 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.513315 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.513523 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:01:08.514468 29808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:08.599371 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29604
I20260809 15:01:08.624599 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29821
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[29209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:08.638412 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:08.625283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.27.186.131:36125)
0809 15:01:08.625481 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:08.625488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:08.625507 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:08.625642 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:08.625646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:08.625721 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:08.625804 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:08.627079 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.627789 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.629059 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:08.629468 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:08.629579 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:08.633338 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:08.633352 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:08.633367 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:08.633404 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:08.635855 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.636474 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.636482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.636484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.636621 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:08.636972 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:08.636976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:08.636978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:08.637314 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:08.637577 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:08.637865 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 15:01:08.638098 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:01:08.639004 30032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:08.724833 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29821
I20260809 15:01:08.745296 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29228
I20260809 15:01:08.746493 29333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:08.849714 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29228
2026-08-09T15:01:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4139 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[30120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[30120](info): set up 2 sockets
Aug 09 15:01:08 dist-test-slave-q9h9 krb5kdc[30120](info): commencing operation
krb5kdc: starting...
W20260809 15:01:10.044724 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:10 dist-test-slave-q9h9 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:10.211057 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37209
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:10.511875 30144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:10.512148 30144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:10.512291 30144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:10.519465 30144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:10.519562 30144 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 15:01:10.519629 30144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:10.519711 30144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:10.519779 30144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:10.519857 30144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:10.528910 30144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:37209
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:10.529948 30144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:10.531138 30144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:10.542066 30150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:10.542047 30152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:10.542109 30154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:10.543365 30144 server_base.cc:1034] running on GCE node
I20260809 15:01:10.544013 30144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:10.545231 30144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:10.546427 30144 hybrid_clock.cc:648] HybridClock initialized: now 1786287670546370 us; error 34 us; skew 500 ppm
I20260809 15:01:10.546711 30144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:10 dist-test-slave-q9h9 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287670, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:10.549484 30144 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:10.550873 30144 webserver.cc:460] Webserver started at http://127.27.186.190:33583/ using document root <none> and password file <none>
I20260809 15:01:10.551489 30144 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:10.551585 30144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:10.551872 30144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:10.554553 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "42a8152054a1466696d4a36af97a931f"
format_stamp: "Formatted at 2026-08-09 15:01:10 on dist-test-slave-q9h9"
server_key: "e3f56ee4134ae91fac11d014a401ab35"
server_key_iv: "0fc83bf47c5cbb0eeaa77a300aa41455"
server_key_version: "encryptionkey@0"
I20260809 15:01:10.555262 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "42a8152054a1466696d4a36af97a931f"
format_stamp: "Formatted at 2026-08-09 15:01:10 on dist-test-slave-q9h9"
server_key: "e3f56ee4134ae91fac11d014a401ab35"
server_key_iv: "0fc83bf47c5cbb0eeaa77a300aa41455"
server_key_version: "encryptionkey@0"
I20260809 15:01:10.559844 30144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:10.562981 30166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:10.564169 30144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:01:10.564329 30144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "42a8152054a1466696d4a36af97a931f"
format_stamp: "Formatted at 2026-08-09 15:01:10 on dist-test-slave-q9h9"
server_key: "e3f56ee4134ae91fac11d014a401ab35"
server_key_iv: "0fc83bf47c5cbb0eeaa77a300aa41455"
server_key_version: "encryptionkey@0"
I20260809 15:01:10.564515 30144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:10.587832 30144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:10.591431 30144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:10.591646 30144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:10.611141 30144 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:37209
I20260809 15:01:10.611161 30269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:37209 every 8 connection(s)
I20260809 15:01:10.612308 30144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:10.614578 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30144
I20260809 15:01:10.614813 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:10.615639 28394 external_mini_cluster.cc:1428] Setting key c9df44ce3960c335863bfa3e8e2b811f
I20260809 15:01:10.616166 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:10.627488 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: Bootstrap starting.
Aug 09 15:01:10 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:10.630841 30272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:10.631908 30272 log.cc:824] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:10.634800 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: No bootstrap required, opened a new log
I20260809 15:01:10.636873 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:10.617571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.27.186.190:37209)
0809 15:01:10.618272 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:10.618285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:10.618321 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:01:10.620413 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:10.620433 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:10.620767 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:10.621110 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:10.622509 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:10.623715 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:10.624908 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:10.625644 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:10.625822 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:10.629952 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:10.629959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:10.629975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:10.630006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:10.633134 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:10.633857 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:10.633862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:10.633867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:10.634065 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:10.634418 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:10.634421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:10.634423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:10.634908 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:10.635186 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:10.635606 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 15:01:10.635975 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":301,"threads_started":1}
I20260809 15:01:10.641963 30272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a8152054a1466696d4a36af97a931f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37209 } }
I20260809 15:01:10.642299 30272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:10.642388 30272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a8152054a1466696d4a36af97a931f, State: Initialized, Role: FOLLOWER
I20260809 15:01:10.642884 30272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a8152054a1466696d4a36af97a931f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37209 } }
I20260809 15:01:10.643098 30272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:10.643229 30272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:10.643373 30272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:10.645556 30272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a8152054a1466696d4a36af97a931f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37209 } }
I20260809 15:01:10.646044 30272 leader_election.cc:302] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42a8152054a1466696d4a36af97a931f; no voters:
I20260809 15:01:10.646535 30272 leader_election.cc:288] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:10.646648 30280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:10.647255 30280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [term 1 LEADER]: Becoming Leader. State: Replica: 42a8152054a1466696d4a36af97a931f, State: Running, Role: LEADER
I20260809 15:01:10.647756 30280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a8152054a1466696d4a36af97a931f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37209 } }
I20260809 15:01:10.648429 30272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:10.650514 30283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42a8152054a1466696d4a36af97a931f. Latest consensus state: current_term: 1 leader_uuid: "42a8152054a1466696d4a36af97a931f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a8152054a1466696d4a36af97a931f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37209 } } }
I20260809 15:01:10.650748 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42a8152054a1466696d4a36af97a931f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a8152054a1466696d4a36af97a931f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37209 } } }
I20260809 15:01:10.651010 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:10.651010 30283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:10.655539 30294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:10.660123 30294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:10.668052 30294 catalog_manager.cc:1347] Generated new cluster ID: 2a72ce1378cc4210bef1d0d172230a3d
I20260809 15:01:10.668172 30294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:10.693847 30294 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:10.695133 30294 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:10.711858 30294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: Generated new TSK 0
I20260809 15:01:10.712553 30294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:10.763407 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37209
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:11.075373 30322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:11.075630 30322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:11.075768 30322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:11.082636 30322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:11.082741 30322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:11.082861 30322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:01:11.082969 30322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:11.092295 30322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:11.093365 30322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:11.094574 30322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:11.106572 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:11.106547 30338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:11.106910 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:11.108587 30322 server_base.cc:1034] running on GCE node
I20260809 15:01:11.109131 30322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:11.109965 30322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:11.111212 30322 hybrid_clock.cc:648] HybridClock initialized: now 1786287671111144 us; error 29 us; skew 500 ppm
I20260809 15:01:11.111472 30322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:11 dist-test-slave-q9h9 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:11.114439 30322 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:11.115723 30322 webserver.cc:460] Webserver started at http://127.27.186.129:33437/ using document root <none> and password file <none>
I20260809 15:01:11.116333 30322 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:11.116427 30322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:11.116719 30322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:11.119414 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "ca015c5195454c989d61cbe384d3f585"
format_stamp: "Formatted at 2026-08-09 15:01:11 on dist-test-slave-q9h9"
server_key: "7df0c8985b26b999ef6a684c66259f64"
server_key_iv: "5049198a5b516382194775ae08a12837"
server_key_version: "encryptionkey@0"
I20260809 15:01:11.120133 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "ca015c5195454c989d61cbe384d3f585"
format_stamp: "Formatted at 2026-08-09 15:01:11 on dist-test-slave-q9h9"
server_key: "7df0c8985b26b999ef6a684c66259f64"
server_key_iv: "5049198a5b516382194775ae08a12837"
server_key_version: "encryptionkey@0"
I20260809 15:01:11.124794 30322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:11.127944 30347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:11.129081 30322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:11.129237 30322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "ca015c5195454c989d61cbe384d3f585"
format_stamp: "Formatted at 2026-08-09 15:01:11 on dist-test-slave-q9h9"
server_key: "7df0c8985b26b999ef6a684c66259f64"
server_key_iv: "5049198a5b516382194775ae08a12837"
server_key_version: "encryptionkey@0"
I20260809 15:01:11.129438 30322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:11.159976 30322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:11.163440 30322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:11.163669 30322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:11.166035 30322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:11.166128 30322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:11.166257 30322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:11.166320 30322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:01:11 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:11.188616 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.166687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:33969 (local address 127.27.186.190:37209)
0809 15:01:11.167153 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:11.167159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:11.168246 (+ 1087us) server_negotiation.cc:407] Connection header received
0809 15:01:11.170760 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:11.170766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:11.170845 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:11.170935 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:11.173184 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.174103 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.176895 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.177385 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.177546 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.182148 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:11.182153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:11.182156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:11.182187 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:11.185361 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.186344 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.186350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.186353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.186496 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.187030 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.187033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.187035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.187297 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:11.187404 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:11.188208 (+ 804us) server_negotiation.cc:299] Negotiation successful
0809 15:01:11.188362 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
I20260809 15:01:11.190008 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.167119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.27.186.129:33969)
0809 15:01:11.167940 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:11.167985 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:11.170433 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:11.171207 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:11.171223 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:11.172045 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:11.172990 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.173018 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.174282 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.174292 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.176746 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.176762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.177691 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.177703 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.178036 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.178785 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:11.178819 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:11.181914 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:11.185526 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.185535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.185549 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.186201 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.186621 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.186626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.186633 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.186916 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.187412 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:11.187420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:11.187846 (+ 426us) client_negotiation.cc:769] Sending connection context
0809 15:01:11.188130 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 15:01:11.188621 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":358,"threads_started":1}
I20260809 15:01:11.209007 30322 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:45607
I20260809 15:01:11.209120 30549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:45607 every 8 connection(s)
I20260809 15:01:11.210453 30322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:11.211521 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.197546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.27.186.129:57181)
0809 15:01:11.198292 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:11.198314 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:11.198452 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:11.198814 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:11.198818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:11.199444 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:11.199881 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.199891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.201209 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.201214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.202936 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.202953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.203917 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.203924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.204045 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.204581 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:11.204597 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:11.205239 (+ 642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:11.208773 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.208782 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.208786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.209354 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.209743 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.209746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.209748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.209859 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.210461 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:11.210464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:11.210630 (+ 166us) client_negotiation.cc:769] Sending connection context
0809 15:01:11.210764 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 15:01:11.211069 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":393,"threads_started":1}
I20260809 15:01:11.211704 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.197574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:57181 (local address 127.27.186.190:37209)
0809 15:01:11.197739 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:11.197746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:11.198397 (+ 651us) server_negotiation.cc:407] Connection header received
0809 15:01:11.198580 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:11.198585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:11.198661 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:11.198754 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:11.200020 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.201079 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.203104 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.203718 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.203851 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.205420 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:11.205427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:11.205433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:11.205468 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:11.208656 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.209475 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.209481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.209485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.209637 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.209961 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.209966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.209969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.210351 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:11.210500 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:11.211117 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 15:01:11.211359 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:01:11.212867 30550 heartbeater.cc:347] Connected to a master server at 127.27.186.190:37209
I20260809 15:01:11.213272 30550 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:11.213963 30550 heartbeater.cc:511] Master 127.27.186.190:37209 requested a full tablet report, sending...
I20260809 15:01:11.215992 30198 ts_manager.cc:194] Registered new tserver with Master: ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607)
I20260809 15:01:11.217284 30198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:33969
I20260809 15:01:11.218616 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30322
I20260809 15:01:11.218847 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:11.219635 28394 external_mini_cluster.cc:1428] Setting key 57dae2b2710c93b3c54042664c0fb54e
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:11.256414 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37209
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:11.560876 30555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:11.561120 30555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:11.561264 30555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:11.568328 30555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:11.568429 30555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:11.568547 30555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:01:11.568652 30555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:11.577713 30555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:11.578811 30555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:11.580003 30555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:11.591626 30561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:11.591737 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:11.591707 30566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:11.592705 30555 server_base.cc:1034] running on GCE node
I20260809 15:01:11.593281 30555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:11.594113 30555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:11.595312 30555 hybrid_clock.cc:648] HybridClock initialized: now 1786287671595243 us; error 53 us; skew 500 ppm
I20260809 15:01:11.595566 30555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:11 dist-test-slave-q9h9 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:11.603474 30555 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:11.604816 30555 webserver.cc:460] Webserver started at http://127.27.186.130:37897/ using document root <none> and password file <none>
I20260809 15:01:11.605441 30555 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:11.605540 30555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:11.605837 30555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:11.608500 30555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "4608c5fbd2a240bebffbe55bbc8b1acd"
format_stamp: "Formatted at 2026-08-09 15:01:11 on dist-test-slave-q9h9"
server_key: "36af09a34882ea179795e4b919c1ec6f"
server_key_iv: "fb3dd1117fb10dd57b67461cec78346c"
server_key_version: "encryptionkey@0"
I20260809 15:01:11.609232 30555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "4608c5fbd2a240bebffbe55bbc8b1acd"
format_stamp: "Formatted at 2026-08-09 15:01:11 on dist-test-slave-q9h9"
server_key: "36af09a34882ea179795e4b919c1ec6f"
server_key_iv: "fb3dd1117fb10dd57b67461cec78346c"
server_key_version: "encryptionkey@0"
I20260809 15:01:11.614189 30555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 15:01:11.617449 30582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:11.618710 30555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:11.618854 30555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "4608c5fbd2a240bebffbe55bbc8b1acd"
format_stamp: "Formatted at 2026-08-09 15:01:11 on dist-test-slave-q9h9"
server_key: "36af09a34882ea179795e4b919c1ec6f"
server_key_iv: "fb3dd1117fb10dd57b67461cec78346c"
server_key_version: "encryptionkey@0"
I20260809 15:01:11.619035 30555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:11.637339 30555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:11.640744 30555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:11.640962 30555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:11.643426 30555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:11.643517 30555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:11.643694 30555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:11.643783 30555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:01:11 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:11.666409 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.643780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:45659 (local address 127.27.186.190:37209)
0809 15:01:11.643965 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:11.643972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:11.645227 (+ 1255us) server_negotiation.cc:407] Connection header received
0809 15:01:11.647425 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:11.647432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:11.647517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:11.647639 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:11.649470 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.650650 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.653065 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.653703 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.653835 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.659813 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:11.659819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:11.659824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:11.659868 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:11.663071 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.663974 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.663978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.663981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.664096 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.664664 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.664669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.664673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.664993 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:11.665150 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:11.665977 (+ 827us) server_negotiation.cc:299] Negotiation successful
0809 15:01:11.666166 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:01:11.667872 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.644226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.27.186.130:45659)
0809 15:01:11.645050 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:11.645091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:11.647135 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:11.647852 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:11.647860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:11.648625 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:11.649322 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.649339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.650811 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.650820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.652910 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.652924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.653957 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.653965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.654302 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.656507 (+ 2205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:11.656546 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:11.659580 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:11.663246 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.663256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.663272 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.663863 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.664217 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.664221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.664223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.664503 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.665153 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:11.665162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:11.665608 (+ 446us) client_negotiation.cc:769] Sending connection context
0809 15:01:11.665918 (+ 310us) client_negotiation.cc:240] Negotiation successful
0809 15:01:11.666439 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":314,"threads_started":1}
I20260809 15:01:11.686143 30555 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:34099
I20260809 15:01:11.686240 30780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:34099 every 8 connection(s)
I20260809 15:01:11.687474 30555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:11.688834 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30555
I20260809 15:01:11.689029 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:11.689589 28394 external_mini_cluster.cc:1428] Setting key 1c85238962a8c03dbdbfce9333ebc645
I20260809 15:01:11.690984 30782 heartbeater.cc:347] Connected to a master server at 127.27.186.190:37209
I20260809 15:01:11.692385 30782 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:11.693110 30782 heartbeater.cc:511] Master 127.27.186.190:37209 requested a full tablet report, sending...
I20260809 15:01:11.694870 30197 ts_manager.cc:194] Registered new tserver with Master: 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099)
I20260809 15:01:11.695955 30197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:45659
I20260809 15:01:11.698495 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.684455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.27.186.130:51173)
0809 15:01:11.685159 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:11.685176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:11.685369 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:11.685719 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:11.685724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:11.686372 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:11.686665 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.686674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.687945 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.687949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.689115 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:11.689130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.690299 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.690307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:11.690504 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.691133 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:11.691158 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:11.692038 (+ 880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:11.695695 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.695700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.695703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.696149 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.696588 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:11.696593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:11.696597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:11.696758 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:11.697447 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:11.697452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:11.697638 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 15:01:11.697775 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 15:01:11.698071 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":391,"threads_started":1}
I20260809 15:01:11.699270 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:11.684374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:51173 (local address 127.27.186.190:37209)
0809 15:01:11.684522 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:11.684525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:11.685258 (+ 733us) server_negotiation.cc:407] Connection header received
0809 15:01:11.685494 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:11.685500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:11.685579 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:11.685684 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:11.686797 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.687824 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.689254 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:11.689896 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:11.690040 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:11.692247 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:11.692253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:11.692258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:11.692294 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:11.695544 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.696278 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.696284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.696288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.696443 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:11.696909 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:11.696914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:11.696916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:11.697309 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:11.698817 (+ 1508us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:11.698908 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 15:01:11.699055 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:11.728307 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37209
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:12.030905 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:12.031136 30789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:12.031263 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:12.038203 30789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:12.038292 30789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:12.038414 30789 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:01:12.038514 30789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:12.047636 30789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:12.048708 30789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:12.049980 30789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:12.061874 30797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:12.062001 30804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:12.061976 30800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:12.063289 30789 server_base.cc:1034] running on GCE node
I20260809 15:01:12.063841 30789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:12.064775 30789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:12.066002 30789 hybrid_clock.cc:648] HybridClock initialized: now 1786287672065932 us; error 40 us; skew 500 ppm
I20260809 15:01:12.066327 30789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:12 dist-test-slave-q9h9 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:12.069434 30789 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:12.070883 30789 webserver.cc:460] Webserver started at http://127.27.186.131:38567/ using document root <none> and password file <none>
I20260809 15:01:12.071506 30789 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:12.071595 30789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:12.071884 30789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:12.074597 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "8c2cc87f076343d8ad826195b9d0789c"
format_stamp: "Formatted at 2026-08-09 15:01:12 on dist-test-slave-q9h9"
server_key: "c04b9b7c8274852fcae1ca8792aeb23a"
server_key_iv: "7c3715abf7975fa46cc21753883cd10a"
server_key_version: "encryptionkey@0"
I20260809 15:01:12.075357 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "8c2cc87f076343d8ad826195b9d0789c"
format_stamp: "Formatted at 2026-08-09 15:01:12 on dist-test-slave-q9h9"
server_key: "c04b9b7c8274852fcae1ca8792aeb23a"
server_key_iv: "7c3715abf7975fa46cc21753883cd10a"
server_key_version: "encryptionkey@0"
I20260809 15:01:12.080097 30789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:12.083267 30812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:12.084415 30789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:01:12.084561 30789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "8c2cc87f076343d8ad826195b9d0789c"
format_stamp: "Formatted at 2026-08-09 15:01:12 on dist-test-slave-q9h9"
server_key: "c04b9b7c8274852fcae1ca8792aeb23a"
server_key_iv: "7c3715abf7975fa46cc21753883cd10a"
server_key_version: "encryptionkey@0"
I20260809 15:01:12.084745 30789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:12.146994 30789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:12.151386 30789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:12.151643 30789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:12.154321 30789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:12.154407 30789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:12.154539 30789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:12.154597 30789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:01:12 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:12.177093 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.154994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:59071 (local address 127.27.186.190:37209)
0809 15:01:12.155191 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.155198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.156433 (+ 1235us) server_negotiation.cc:407] Connection header received
0809 15:01:12.159300 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.159304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.159382 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.159493 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:12.161746 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.162650 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.165446 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.165909 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.166063 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.170860 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:12.170867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:12.170872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:12.170942 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:12.173841 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.174771 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.174777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.174781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.174925 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.175351 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.175355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.175357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.175633 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:12.175778 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.176496 (+ 718us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.176740 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:01:12.178170 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.155469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.27.186.131:59071)
0809 15:01:12.156250 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.156311 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.158992 (+ 2681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.159768 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.159796 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.160609 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:12.161551 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.161576 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.162860 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.162866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.165308 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.165323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.166187 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.166194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.166521 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.167292 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:12.167322 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:12.170593 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:12.173963 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:12.173972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:12.173986 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:12.174638 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:12.175031 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:12.175036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:12.175040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:12.175222 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:12.175757 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:12.175764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:12.176082 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.176316 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.176773 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":316,"threads_started":1}
I20260809 15:01:12.197053 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.183841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:46823 (local address 127.27.186.190:37209)
0809 15:01:12.184014 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.184019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.184701 (+ 682us) server_negotiation.cc:407] Connection header received
0809 15:01:12.184982 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.184987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.185057 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.185111 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:12.186456 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.187363 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.188835 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.189515 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.189642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.191753 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:12.191759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:12.191763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:12.191792 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:12.194594 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.195256 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.195262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.195265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.195385 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.195709 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.195714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.195718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.195983 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:12.196115 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.196519 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.196710 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 15:01:12.197147 30935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.183837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.27.186.131:46823)
0809 15:01:12.184609 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.184625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.184824 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.185285 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.185290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.185810 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:12.186320 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.186334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.187539 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.187544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.188638 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.188650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.189681 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.189686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.189807 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.190395 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:12.190413 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:12.191504 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:12.194720 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:12.194725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:12.194729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:12.195128 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:12.195488 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:12.195493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:12.195497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:12.195614 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:12.196093 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:12.196098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:12.196266 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.196382 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.196690 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":472,"threads_started":1}
I20260809 15:01:12.207417 30789 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:37027
I20260809 15:01:12.207456 31025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:37027 every 8 connection(s)
I20260809 15:01:12.208695 30789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:12.210665 31027 heartbeater.cc:347] Connected to a master server at 127.27.186.190:37209
I20260809 15:01:12.210883 31027 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:12.211474 31027 heartbeater.cc:511] Master 127.27.186.190:37209 requested a full tablet report, sending...
I20260809 15:01:12.212011 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30789
I20260809 15:01:12.212203 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:12.212690 28394 external_mini_cluster.cc:1428] Setting key ea61b156a85eaf05e0cbe0adb8849810
I20260809 15:01:12.213020 30197 ts_manager.cc:194] Registered new tserver with Master: 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027)
I20260809 15:01:12.214128 30197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:59071
I20260809 15:01:12.215641 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:01:12.220031 30550 heartbeater.cc:503] Master 127.27.186.190:37209 was elected leader, sending a full tablet report...
Aug 09 15:01:12 dist-test-slave-q9h9 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 15:01:12 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:12.261173 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.248467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.27.186.190:37209)
0809 15:01:12.248618 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.248623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.248922 (+ 299us) server_negotiation.cc:407] Connection header received
0809 15:01:12.249088 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.249107 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.249199 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.249287 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:12.250649 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.251383 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.252598 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.253061 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.253155 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.256775 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:12.256781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:12.256785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:12.256825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:12.259093 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.259640 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.259644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.259646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.259759 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.260105 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.260108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.260110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.260363 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:12.260545 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.260733 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.260891 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:01:12.264832 30197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:12.271803 30197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:12.310431 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.284067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.27.186.130:34099)
0809 15:01:12.301417 (+ 17350us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.301426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.301458 (+ 32us) server_negotiation.cc:407] Connection header received
0809 15:01:12.301565 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.301573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.301842 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.302071 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.302791 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.304388 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.307110 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.308535 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.308706 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.308912 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.309948 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.310142 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17172,"thread_start_us":18174,"threads_started":1}
I20260809 15:01:12.309731 31050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.283803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34099 (local address 127.0.0.1:58854)
0809 15:01:12.299494 (+ 15691us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.299548 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.299719 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.302093 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.302101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.302135 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.302596 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.302608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.304531 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.304537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.306938 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.306954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.308754 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.308760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.308977 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.309034 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.309154 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.309343 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15422,"thread_start_us":16077,"threads_started":1}
I20260809 15:01:12.314241 30666 tablet_service.cc:1467] Processing CreateTablet for tablet f6a5530d1c174a5283b5826f9ca1ceed (DEFAULT_TABLE table=test-table [id=a572c6cea5c043d7b82e57302d8b42eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:12.315762 30666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a5530d1c174a5283b5826f9ca1ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.316085 31051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.283901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:37027 (local address 127.0.0.1:40678)
0809 15:01:12.308601 (+ 24700us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.308621 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.308741 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.309765 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.309770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.309795 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.310285 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.310298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.312189 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.312194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.314047 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.314063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.315373 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.315379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.315533 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.315566 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.315637 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.315762 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24608,"thread_start_us":25054,"threads_started":1}
I20260809 15:01:12.316183 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.284596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.27.186.131:37027)
0809 15:01:12.308173 (+ 23577us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.308178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.308870 (+ 692us) server_negotiation.cc:407] Connection header received
0809 15:01:12.308975 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.308981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.309209 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.309400 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.310493 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.312030 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.314215 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.315174 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.315350 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.315510 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.315718 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.315884 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23437,"thread_start_us":23557,"threads_started":1}
I20260809 15:01:12.318696 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.284322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45607 (local address 127.0.0.1:34372)
0809 15:01:12.310345 (+ 26023us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.310367 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.310562 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.311386 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.311391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.311415 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.311786 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.311796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.313554 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.313559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.316643 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.316658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.317941 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.317947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.318145 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.318176 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.318265 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.318393 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25901,"thread_start_us":25750,"threads_started":1}
I20260809 15:01:12.318792 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.284647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.27.186.129:45607)
0809 15:01:12.293546 (+ 8899us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.293556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.310460 (+ 16904us) server_negotiation.cc:407] Connection header received
0809 15:01:12.310751 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.310759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.310989 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.311138 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.311973 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.313413 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.316793 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.317720 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.317914 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.318110 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.318337 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.318501 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8723,"thread_start_us":8571,"threads_started":1}
I20260809 15:01:12.320230 30896 tablet_service.cc:1467] Processing CreateTablet for tablet f6a5530d1c174a5283b5826f9ca1ceed (DEFAULT_TABLE table=test-table [id=a572c6cea5c043d7b82e57302d8b42eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:12.321344 30896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a5530d1c174a5283b5826f9ca1ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.321668 30424 tablet_service.cc:1467] Processing CreateTablet for tablet f6a5530d1c174a5283b5826f9ca1ceed (DEFAULT_TABLE table=test-table [id=a572c6cea5c043d7b82e57302d8b42eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:12.322757 30424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a5530d1c174a5283b5826f9ca1ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.332933 31058 tablet_bootstrap.cc:492] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Bootstrap starting.
I20260809 15:01:12.335830 31063 tablet_bootstrap.cc:492] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Bootstrap starting.
I20260809 15:01:12.336283 31058 tablet_bootstrap.cc:654] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.337180 31060 tablet_bootstrap.cc:492] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Bootstrap starting.
I20260809 15:01:12.338240 31058 log.cc:824] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:12.339188 31063 tablet_bootstrap.cc:654] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.340297 31063 log.cc:824] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:12.340413 31060 tablet_bootstrap.cc:654] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.341452 31058 tablet_bootstrap.cc:492] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: No bootstrap required, opened a new log
I20260809 15:01:12.341560 31060 log.cc:824] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:12.341727 31058 ts_tablet_manager.cc:1397] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260809 15:01:12.343291 31063 tablet_bootstrap.cc:492] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: No bootstrap required, opened a new log
I20260809 15:01:12.343547 31063 ts_tablet_manager.cc:1397] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 15:01:12.343936 31060 tablet_bootstrap.cc:492] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: No bootstrap required, opened a new log
I20260809 15:01:12.344158 31060 ts_tablet_manager.cc:1397] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:01:12.351787 31060 raft_consensus.cc:348] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.351824 31058 raft_consensus.cc:348] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.352188 31060 raft_consensus.cc:374] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.352276 31060 raft_consensus.cc:729] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2cc87f076343d8ad826195b9d0789c, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.352298 31058 raft_consensus.cc:374] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.352430 31058 raft_consensus.cc:729] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca015c5195454c989d61cbe384d3f585, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.352852 31060 consensus_queue.cc:260] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.353029 31058 consensus_queue.cc:260] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.353565 31063 raft_consensus.cc:348] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.353863 31027 heartbeater.cc:503] Master 127.27.186.190:37209 was elected leader, sending a full tablet report...
I20260809 15:01:12.354094 31063 raft_consensus.cc:374] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.354249 31063 raft_consensus.cc:729] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4608c5fbd2a240bebffbe55bbc8b1acd, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.354544 31060 ts_tablet_manager.cc:1428] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260809 15:01:12.354976 31063 consensus_queue.cc:260] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.356097 31058 ts_tablet_manager.cc:1428] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260809 15:01:12.356923 30782 heartbeater.cc:503] Master 127.27.186.190:37209 was elected leader, sending a full tablet report...
I20260809 15:01:12.358171 31063 ts_tablet_manager.cc:1428] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260809 15:01:12.439410 31072 raft_consensus.cc:482] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:12.439682 31072 raft_consensus.cc:504] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.441229 31072 leader_election.cc:288] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099), 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027)
W20260809 15:01:12.442282 30783 tablet.cc:2369] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:12.449697 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.441871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:39175 (local address 127.27.186.131:37027)
0809 15:01:12.442103 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.442109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.443062 (+ 953us) server_negotiation.cc:407] Connection header received
0809 15:01:12.443243 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.443248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.443327 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.443400 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.444000 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.444989 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.447508 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.448695 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.448860 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.448930 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.449322 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.449448 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:01:12.449750 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.441637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:34999 (local address 127.27.186.130:34099)
0809 15:01:12.441831 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.441836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.442735 (+ 899us) server_negotiation.cc:407] Connection header received
0809 15:01:12.443061 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.443067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.443138 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.443221 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.443787 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.444850 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.447514 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.448701 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.448867 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.448930 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.449408 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.449526 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:01:12.449808 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.441799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:37027 (local address 127.27.186.129:39175)
0809 15:01:12.442966 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.442984 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.443118 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.443440 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.443444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.443466 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.443851 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.443860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.445102 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.445107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.447355 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.447371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.448901 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.448907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.449091 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.449128 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.449269 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.449406 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":7,"thread_start_us":479,"threads_started":1}
I20260809 15:01:12.450421 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.441691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34099 (local address 127.27.186.129:34999)
0809 15:01:12.442628 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.442649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.442862 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.443248 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.443252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.443272 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.443627 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.443639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.444970 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.444973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.447355 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.447371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.449074 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.449079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.449227 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.449256 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.449916 (+ 660us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.450071 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":83,"thread_start_us":999,"threads_started":1}
I20260809 15:01:12.450814 30942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a5530d1c174a5283b5826f9ca1ceed" candidate_uuid: "ca015c5195454c989d61cbe384d3f585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2cc87f076343d8ad826195b9d0789c" is_pre_election: true
I20260809 15:01:12.451377 30942 raft_consensus.cc:2454] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca015c5195454c989d61cbe384d3f585 in term 0.
I20260809 15:01:12.451387 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a5530d1c174a5283b5826f9ca1ceed" candidate_uuid: "ca015c5195454c989d61cbe384d3f585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" is_pre_election: true
I20260809 15:01:12.451880 30703 raft_consensus.cc:2454] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca015c5195454c989d61cbe384d3f585 in term 0.
I20260809 15:01:12.452069 30352 leader_election.cc:302] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c2cc87f076343d8ad826195b9d0789c, ca015c5195454c989d61cbe384d3f585; no voters:
I20260809 15:01:12.452490 31072 raft_consensus.cc:2781] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:12.452633 31072 raft_consensus.cc:482] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:12.452729 31072 raft_consensus.cc:3037] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.454871 31072 raft_consensus.cc:504] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
I20260809 15:01:12.455384 31072 leader_election.cc:288] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [CANDIDATE]: Term 1 election: Requested vote from peers 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099), 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027)
I20260809 15:01:12.455744 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a5530d1c174a5283b5826f9ca1ceed" candidate_uuid: "ca015c5195454c989d61cbe384d3f585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd"
I20260809 15:01:12.455833 30942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a5530d1c174a5283b5826f9ca1ceed" candidate_uuid: "ca015c5195454c989d61cbe384d3f585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2cc87f076343d8ad826195b9d0789c"
I20260809 15:01:12.455909 30703 raft_consensus.cc:3037] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.456024 30942 raft_consensus.cc:3037] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.457947 30703 raft_consensus.cc:2454] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca015c5195454c989d61cbe384d3f585 in term 1.
I20260809 15:01:12.458328 30349 leader_election.cc:302] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4608c5fbd2a240bebffbe55bbc8b1acd, ca015c5195454c989d61cbe384d3f585; no voters:
I20260809 15:01:12.458528 30942 raft_consensus.cc:2454] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca015c5195454c989d61cbe384d3f585 in term 1.
I20260809 15:01:12.458626 31072 raft_consensus.cc:2781] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:12.459088 31072 raft_consensus.cc:686] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 1 LEADER]: Becoming Leader. State: Replica: ca015c5195454c989d61cbe384d3f585, State: Running, Role: LEADER
I20260809 15:01:12.459548 31072 consensus_queue.cc:237] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } }
W20260809 15:01:12.460153 31033 tablet.cc:2369] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:01:12.463527 30551 tablet.cc:2369] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:12.463610 30198 catalog_manager.cc:5495] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca015c5195454c989d61cbe384d3f585 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "ca015c5195454c989d61cbe384d3f585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } health_report { overall_health: HEALTHY } } }
I20260809 15:01:12.514575 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.509699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.27.186.129:45607)
0809 15:01:12.509859 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.509863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.510066 (+ 203us) server_negotiation.cc:407] Connection header received
0809 15:01:12.510379 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.510383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.510467 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.510578 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:12.510870 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.511529 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.512770 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.513150 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.513235 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.513640 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:12.514157 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:12.514264 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.514329 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.514391 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:01:12.522763 30942 raft_consensus.cc:1261] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Refusing update from remote peer ca015c5195454c989d61cbe384d3f585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:12.523928 31080 consensus_queue.cc:1035] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:12.524130 30703 raft_consensus.cc:1261] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Refusing update from remote peer ca015c5195454c989d61cbe384d3f585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:12.524832 31080 consensus_queue.cc:1035] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:12.534045 31087 mvcc.cc:205] Tried to move back new op lower bound from 7316634306641661952 to 7316634306397978624. Current Snapshot: MvccSnapshot[applied={T|T < 7316634306641661952}]
I20260809 15:01:12.536782 31088 mvcc.cc:205] Tried to move back new op lower bound from 7316634306641661952 to 7316634306397978624. Current Snapshot: MvccSnapshot[applied={T|T < 7316634306641661952}]
I20260809 15:01:12.555438 30198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:12.555717 30198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:12.559803 30198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: Sending DeleteTablet for 3 replicas of tablet f6a5530d1c174a5283b5826f9ca1ceed
I20260809 15:01:12.561170 30666 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a5530d1c174a5283b5826f9ca1ceed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:12 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:58854
I20260809 15:01:12.562239 30424 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a5530d1c174a5283b5826f9ca1ceed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:12 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34372
I20260809 15:01:12.562832 30896 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a5530d1c174a5283b5826f9ca1ceed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:12 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:40678
I20260809 15:01:12.562855 30198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:12.563432 30198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:12.564339 31102 tablet_replica.cc:330] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: stopping tablet replica
I20260809 15:01:12.564975 31105 tablet_replica.cc:330] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: stopping tablet replica
I20260809 15:01:12.564917 31102 raft_consensus.cc:2229] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:12.565331 31105 raft_consensus.cc:2229] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:12.565562 31102 raft_consensus.cc:2258] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:12.565779 31105 raft_consensus.cc:2258] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:12.566489 31103 tablet_replica.cc:330] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: stopping tablet replica
I20260809 15:01:12.566890 31103 raft_consensus.cc:2229] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:12.567771 31103 raft_consensus.cc:2258] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:12.568250 31102 ts_tablet_manager.cc:1905] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:12.570658 30424 tablet_service.cc:1467] Processing CreateTablet for tablet 39845f1af4974d37a14858af36b068aa (DEFAULT_TABLE table=test-table [id=065c861133fc49b3bdb22af8541e7ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:12.570983 31103 ts_tablet_manager.cc:1905] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:12.571135 30424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39845f1af4974d37a14858af36b068aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.572980 30896 tablet_service.cc:1467] Processing CreateTablet for tablet 39845f1af4974d37a14858af36b068aa (DEFAULT_TABLE table=test-table [id=065c861133fc49b3bdb22af8541e7ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:12.573383 30896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39845f1af4974d37a14858af36b068aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.576558 31105 ts_tablet_manager.cc:1905] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:12.577714 31058 tablet_bootstrap.cc:492] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Bootstrap starting.
I20260809 15:01:12.579334 31058 tablet_bootstrap.cc:654] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.579602 31060 tablet_bootstrap.cc:492] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Bootstrap starting.
I20260809 15:01:12.580899 30666 tablet_service.cc:1467] Processing CreateTablet for tablet 39845f1af4974d37a14858af36b068aa (DEFAULT_TABLE table=test-table [id=065c861133fc49b3bdb22af8541e7ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:12.581117 31060 tablet_bootstrap.cc:654] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.582623 31058 tablet_bootstrap.cc:492] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: No bootstrap required, opened a new log
I20260809 15:01:12.582770 31058 ts_tablet_manager.cc:1397] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260809 15:01:12.583403 31060 tablet_bootstrap.cc:492] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: No bootstrap required, opened a new log
I20260809 15:01:12.583527 31060 ts_tablet_manager.cc:1397] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 15:01:12.583580 31058 raft_consensus.cc:348] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.583817 31058 raft_consensus.cc:374] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.583928 31058 raft_consensus.cc:729] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca015c5195454c989d61cbe384d3f585, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.584188 31060 raft_consensus.cc:348] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.584190 31058 consensus_queue.cc:260] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.584461 31060 raft_consensus.cc:374] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.584604 31060 raft_consensus.cc:729] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2cc87f076343d8ad826195b9d0789c, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.584811 31058 ts_tablet_manager.cc:1428] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 15:01:12.584908 31060 consensus_queue.cc:260] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.585517 31060 ts_tablet_manager.cc:1428] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:12.600684 30666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39845f1af4974d37a14858af36b068aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.607523 31063 tablet_bootstrap.cc:492] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Bootstrap starting.
I20260809 15:01:12.608392 31103 ts_tablet_manager.cc:1918] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:12.608721 31103 log.cc:1196] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/f6a5530d1c174a5283b5826f9ca1ceed
I20260809 15:01:12.609231 31063 tablet_bootstrap.cc:654] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.609506 31103 ts_tablet_manager.cc:1939] T f6a5530d1c174a5283b5826f9ca1ceed P ca015c5195454c989d61cbe384d3f585: Deleting consensus metadata
I20260809 15:01:12.611050 31063 tablet_bootstrap.cc:492] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: No bootstrap required, opened a new log
I20260809 15:01:12.611191 31063 ts_tablet_manager.cc:1397] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260809 15:01:12.611812 31063 raft_consensus.cc:348] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.611992 31063 raft_consensus.cc:374] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.612128 31063 raft_consensus.cc:729] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4608c5fbd2a240bebffbe55bbc8b1acd, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.612377 31063 consensus_queue.cc:260] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.614782 31063 ts_tablet_manager.cc:1428] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260809 15:01:12.648885 31077 raft_consensus.cc:482] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:12.649062 31077 raft_consensus.cc:504] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.650701 31077 leader_election.cc:288] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607), 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027)
I20260809 15:01:12.659333 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.651153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56339 (local address 127.27.186.129:45607)
0809 15:01:12.651368 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.651375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.653409 (+ 2034us) server_negotiation.cc:407] Connection header received
0809 15:01:12.653606 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.653611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.653673 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.653757 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.654345 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.655445 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.657429 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.658340 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.658493 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.658571 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.658904 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.659035 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:01:12.659361 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.650927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45607 (local address 127.27.186.130:56339)
0809 15:01:12.653065 (+ 2138us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.653085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.653466 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.653776 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.653781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.653804 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.654133 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.654146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.655629 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.655634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.657243 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.657257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.658527 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.658533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.658725 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.658761 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.658861 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.658988 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1978,"mutex_wait_us":8,"thread_start_us":2143,"threads_started":1}
I20260809 15:01:12.660480 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.651681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:41053 (local address 127.27.186.131:37027)
0809 15:01:12.651862 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.651867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.653585 (+ 1718us) server_negotiation.cc:407] Connection header received
0809 15:01:12.653850 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.653855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.653929 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.653992 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.654766 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.655827 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.658284 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.659340 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.659479 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.659551 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.660106 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.660240 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:01:12.660526 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.651134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:37027 (local address 127.27.186.130:41053)
0809 15:01:12.653486 (+ 2352us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.653504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.653693 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.654230 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.654234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.654259 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.654611 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.654622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.655988 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.655993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.658130 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.658146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.659758 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.659765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.659942 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.659976 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.660070 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.660188 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2229,"thread_start_us":1580,"threads_started":1}
I20260809 15:01:12.660305 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39845f1af4974d37a14858af36b068aa" candidate_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca015c5195454c989d61cbe384d3f585" is_pre_election: true
I20260809 15:01:12.660791 30459 raft_consensus.cc:2454] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4608c5fbd2a240bebffbe55bbc8b1acd in term 0.
I20260809 15:01:12.661482 30590 leader_election.cc:302] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4608c5fbd2a240bebffbe55bbc8b1acd, ca015c5195454c989d61cbe384d3f585; no voters:
I20260809 15:01:12.661906 31077 raft_consensus.cc:2781] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:12.661834 30942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39845f1af4974d37a14858af36b068aa" candidate_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2cc87f076343d8ad826195b9d0789c" is_pre_election: true
I20260809 15:01:12.662001 31077 raft_consensus.cc:482] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:12.662118 31077 raft_consensus.cc:3037] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.662123 30942 raft_consensus.cc:2454] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4608c5fbd2a240bebffbe55bbc8b1acd in term 0.
I20260809 15:01:12.662835 30171 catalog_manager.cc:4841] TS ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607): tablet f6a5530d1c174a5283b5826f9ca1ceed (table test-table [id=a572c6cea5c043d7b82e57302d8b42eb]) successfully deleted
I20260809 15:01:12.680619 31077 raft_consensus.cc:504] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.681628 31105 ts_tablet_manager.cc:1918] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:12.681926 31105 log.cc:1196] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/f6a5530d1c174a5283b5826f9ca1ceed
I20260809 15:01:12.682718 31105 ts_tablet_manager.cc:1939] T f6a5530d1c174a5283b5826f9ca1ceed P 8c2cc87f076343d8ad826195b9d0789c: Deleting consensus metadata
I20260809 15:01:12.683648 31077 leader_election.cc:288] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [CANDIDATE]: Term 1 election: Requested vote from peers ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607), 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027)
I20260809 15:01:12.684656 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39845f1af4974d37a14858af36b068aa" candidate_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca015c5195454c989d61cbe384d3f585"
I20260809 15:01:12.684911 30459 raft_consensus.cc:3037] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.685380 31102 ts_tablet_manager.cc:1918] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:12.685526 31102 log.cc:1196] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/f6a5530d1c174a5283b5826f9ca1ceed
I20260809 15:01:12.686048 31102 ts_tablet_manager.cc:1939] T f6a5530d1c174a5283b5826f9ca1ceed P 4608c5fbd2a240bebffbe55bbc8b1acd: Deleting consensus metadata
I20260809 15:01:12.689880 30942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39845f1af4974d37a14858af36b068aa" candidate_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2cc87f076343d8ad826195b9d0789c"
I20260809 15:01:12.690124 30942 raft_consensus.cc:3037] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.697469 30172 catalog_manager.cc:4841] TS 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027): tablet f6a5530d1c174a5283b5826f9ca1ceed (table test-table [id=a572c6cea5c043d7b82e57302d8b42eb]) successfully deleted
I20260809 15:01:12.781323 30459 raft_consensus.cc:2454] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4608c5fbd2a240bebffbe55bbc8b1acd in term 1.
I20260809 15:01:12.781942 30590 leader_election.cc:302] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4608c5fbd2a240bebffbe55bbc8b1acd, ca015c5195454c989d61cbe384d3f585; no voters:
I20260809 15:01:12.782127 30168 catalog_manager.cc:4841] TS 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099): tablet f6a5530d1c174a5283b5826f9ca1ceed (table test-table [id=a572c6cea5c043d7b82e57302d8b42eb]) successfully deleted
I20260809 15:01:12.782384 31077 raft_consensus.cc:2781] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:12.782594 30942 raft_consensus.cc:2454] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4608c5fbd2a240bebffbe55bbc8b1acd in term 1.
I20260809 15:01:12.783172 31077 raft_consensus.cc:686] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 LEADER]: Becoming Leader. State: Replica: 4608c5fbd2a240bebffbe55bbc8b1acd, State: Running, Role: LEADER
I20260809 15:01:12.783829 31077 consensus_queue.cc:237] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.787822 30198 catalog_manager.cc:5495] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130). New cstate: current_term: 1 leader_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } health_report { overall_health: UNKNOWN } } }
Aug 09 15:01:12 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287670, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:12.842492 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.827923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37209 (local address 127.0.0.1:40050)
0809 15:01:12.828449 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.828477 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.828629 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.828988 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.828996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.830243 (+ 1247us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:12.830442 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.830452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.831490 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.831494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.832432 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.832444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.833210 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.833215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.833382 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.833941 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:12.833981 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:12.836391 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:12.840131 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:12.840141 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:12.840146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:12.840532 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:12.840946 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:12.840950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:12.840952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:12.841086 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:12.841728 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:12.841733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:12.841870 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.842002 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.842226 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":264,"threads_started":1}
I20260809 15:01:12.842828 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.828028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.27.186.190:37209)
0809 15:01:12.828742 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.828747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.828767 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:12.828807 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.828810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.828859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.828940 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:12.830584 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.831358 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.832557 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.833055 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.833182 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.836593 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:12.836600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:12.836605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:12.836642 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:12.839990 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.840660 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.840666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.840670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.840826 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:12.841213 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:12.841219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:12.841222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:12.841582 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:12.841760 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.842156 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.842392 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":470,"threads_started":1}
I20260809 15:01:12.847756 30198 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:40050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 15:01:12.848991 30198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:12.855769 30424 tablet_service.cc:1467] Processing CreateTablet for tablet ca090fc4b6024705b14a24c38d41c116 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e833063591e847feb20e43cc596c84d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:01:12.855880 30666 tablet_service.cc:1467] Processing CreateTablet for tablet ca090fc4b6024705b14a24c38d41c116 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e833063591e847feb20e43cc596c84d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:01:12.856014 30896 tablet_service.cc:1467] Processing CreateTablet for tablet ca090fc4b6024705b14a24c38d41c116 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e833063591e847feb20e43cc596c84d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:01:12.856391 30424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca090fc4b6024705b14a24c38d41c116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.856422 30666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca090fc4b6024705b14a24c38d41c116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.856491 30896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca090fc4b6024705b14a24c38d41c116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:12.864159 31063 tablet_bootstrap.cc:492] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: Bootstrap starting.
I20260809 15:01:12.864141 31058 tablet_bootstrap.cc:492] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: Bootstrap starting.
I20260809 15:01:12.865876 31058 tablet_bootstrap.cc:654] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.865876 31063 tablet_bootstrap.cc:654] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.868295 31063 tablet_bootstrap.cc:492] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: No bootstrap required, opened a new log
I20260809 15:01:12.868464 31063 ts_tablet_manager.cc:1397] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 15:01:12.869299 31063 raft_consensus.cc:348] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.869576 31063 raft_consensus.cc:374] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.869654 31063 raft_consensus.cc:729] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4608c5fbd2a240bebffbe55bbc8b1acd, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.869905 31063 consensus_queue.cc:260] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.870527 31063 ts_tablet_manager.cc:1428] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:12.870405 31077 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:12.870688 31077 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:12.871699 31060 tablet_bootstrap.cc:492] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: Bootstrap starting.
I20260809 15:01:12.873731 31060 tablet_bootstrap.cc:654] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:12.875890 31060 tablet_bootstrap.cc:492] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: No bootstrap required, opened a new log
I20260809 15:01:12.876065 31060 ts_tablet_manager.cc:1397] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 15:01:12.876847 31060 raft_consensus.cc:348] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.877112 31060 raft_consensus.cc:374] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.877202 31060 raft_consensus.cc:729] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2cc87f076343d8ad826195b9d0789c, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.877449 31060 consensus_queue.cc:260] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.877887 31070 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:12.878202 31070 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:12.879300 31060 ts_tablet_manager.cc:1428] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 15:01:12.885685 31058 tablet_bootstrap.cc:492] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: No bootstrap required, opened a new log
I20260809 15:01:12.885771 31070 raft_consensus.cc:482] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:12.885866 31058 ts_tablet_manager.cc:1397] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.009s
I20260809 15:01:12.885973 31070 raft_consensus.cc:504] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.886538 31058 raft_consensus.cc:348] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.886741 31058 raft_consensus.cc:374] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:12.886832 31058 raft_consensus.cc:729] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca015c5195454c989d61cbe384d3f585, State: Initialized, Role: FOLLOWER
I20260809 15:01:12.887079 31058 consensus_queue.cc:260] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.887670 31058 ts_tablet_manager.cc:1428] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 15:01:12.887521 31080 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:12.887727 31070 leader_election.cc:288] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607), 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099)
I20260809 15:01:12.887841 31080 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:12.898337 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.888437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:52921 (local address 127.27.186.129:45607)
0809 15:01:12.888766 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.888772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.888833 (+ 61us) server_negotiation.cc:407] Connection header received
0809 15:01:12.890454 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.890464 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.890545 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.890607 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.891375 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.892559 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.895124 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.897224 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.897400 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.897478 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.897919 (+ 441us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.898073 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260809 15:01:12.898590 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.888161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45607 (local address 127.27.186.131:52921)
0809 15:01:12.888736 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.888755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.888935 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.890823 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.890829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.890857 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.891182 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.891194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.892767 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.892773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.894937 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.894951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.897580 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.897586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.897754 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.897779 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.897862 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.897982 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":2080,"threads_started":1}
I20260809 15:01:12.899178 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca090fc4b6024705b14a24c38d41c116" candidate_uuid: "8c2cc87f076343d8ad826195b9d0789c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca015c5195454c989d61cbe384d3f585" is_pre_election: true
I20260809 15:01:12.899432 30459 raft_consensus.cc:2454] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c2cc87f076343d8ad826195b9d0789c in term 0.
I20260809 15:01:12.900581 30819 leader_election.cc:302] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c2cc87f076343d8ad826195b9d0789c, ca015c5195454c989d61cbe384d3f585; no voters:
I20260809 15:01:12.900977 31070 raft_consensus.cc:2781] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:12.901121 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.889161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:44953 (local address 127.27.186.130:34099)
0809 15:01:12.889495 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:12.889501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:12.894595 (+ 5094us) server_negotiation.cc:407] Connection header received
0809 15:01:12.894759 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:12.894764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:12.894848 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:12.894940 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:12.895538 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.896676 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.899198 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.900110 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.900238 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.900285 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:12.900726 (+ 441us) server_negotiation.cc:299] Negotiation successful
0809 15:01:12.900848 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260809 15:01:12.901120 31070 raft_consensus.cc:482] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:12.901222 31070 raft_consensus.cc:3037] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.901556 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:12.888759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34099 (local address 127.27.186.131:44953)
0809 15:01:12.894482 (+ 5723us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:12.894501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:12.894629 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:12.894982 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:12.894987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:12.895015 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:12.895369 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.895382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.896853 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.896859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.898970 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:12.898986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:12.900331 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:12.900338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:12.900524 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:12.900559 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 15:01:12.900667 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 15:01:12.900872 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5594,"thread_start_us":5439,"threads_started":1}
I20260809 15:01:12.902297 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca090fc4b6024705b14a24c38d41c116" candidate_uuid: "8c2cc87f076343d8ad826195b9d0789c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" is_pre_election: true
I20260809 15:01:12.902530 30703 raft_consensus.cc:2454] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c2cc87f076343d8ad826195b9d0789c in term 0.
I20260809 15:01:12.985177 31070 raft_consensus.cc:504] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:12.985986 31070 leader_election.cc:288] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [CANDIDATE]: Term 1 election: Requested vote from peers ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607), 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099)
I20260809 15:01:12.986656 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca090fc4b6024705b14a24c38d41c116" candidate_uuid: "8c2cc87f076343d8ad826195b9d0789c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca015c5195454c989d61cbe384d3f585"
I20260809 15:01:12.986907 30459 raft_consensus.cc:3037] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:12.986850 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca090fc4b6024705b14a24c38d41c116" candidate_uuid: "8c2cc87f076343d8ad826195b9d0789c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd"
I20260809 15:01:12.987094 30703 raft_consensus.cc:3037] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:13.090170 30459 raft_consensus.cc:2454] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c2cc87f076343d8ad826195b9d0789c in term 1.
I20260809 15:01:13.090211 30703 raft_consensus.cc:2454] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c2cc87f076343d8ad826195b9d0789c in term 1.
I20260809 15:01:13.090723 30819 leader_election.cc:302] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4608c5fbd2a240bebffbe55bbc8b1acd, 8c2cc87f076343d8ad826195b9d0789c, ca015c5195454c989d61cbe384d3f585; no voters:
I20260809 15:01:13.091060 31070 raft_consensus.cc:2781] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:13.091459 31070 raft_consensus.cc:686] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [term 1 LEADER]: Becoming Leader. State: Replica: 8c2cc87f076343d8ad826195b9d0789c, State: Running, Role: LEADER
I20260809 15:01:13.091895 31070 consensus_queue.cc:237] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } }
I20260809 15:01:13.094796 31135 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: TxnStatusTablet state changed. Reason: New leader 8c2cc87f076343d8ad826195b9d0789c. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.095175 31135 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:01:13.095825 31141 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 15:01:13.096103 31141 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 15:01:13.096436 30198 catalog_manager.cc:5495] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131). New cstate: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } health_report { overall_health: HEALTHY } } }
I20260809 15:01:13.115434 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.109729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.27.186.131:37027)
0809 15:01:13.109891 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.109897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.110138 (+ 241us) server_negotiation.cc:407] Connection header received
0809 15:01:13.110403 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.110407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.110504 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.110627 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:13.110979 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.111710 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.113161 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.113663 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.113782 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.114297 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.114778 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.114922 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.115012 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.115090 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:01:13.115430 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.109810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:37027 (local address 127.0.0.1:40682)
0809 15:01:13.110048 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:13.110068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:13.110207 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:13.110661 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:13.110665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:13.110677 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:13.110853 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:13.110862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.111854 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.111860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:13.113020 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:13.113034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.113819 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.113826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:13.114008 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.114099 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.114892 (+ 793us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.114915 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:01:13.114993 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 15:01:13.115137 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 15:01:13.119902 30703 raft_consensus.cc:1261] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Refusing update from remote peer 8c2cc87f076343d8ad826195b9d0789c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:13.120383 30459 raft_consensus.cc:1261] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Refusing update from remote peer 8c2cc87f076343d8ad826195b9d0789c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:13.120517 31070 consensus_queue.cc:1035] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:13.121078 31135 consensus_queue.cc:1035] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:13.124845 31080 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: TxnStatusTablet state changed. Reason: New leader 8c2cc87f076343d8ad826195b9d0789c. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.125121 31080 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:13.134410 31135 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.134656 31135 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:01:13.136096 31077 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: TxnStatusTablet state changed. Reason: New leader 8c2cc87f076343d8ad826195b9d0789c. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.136371 31077 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:13.137128 31080 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.137362 31080 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P ca015c5195454c989d61cbe384d3f585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:13.137984 31158 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.138468 31158 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 8c2cc87f076343d8ad826195b9d0789c: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:01:13.145418 31077 tablet_replica.cc:439] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c2cc87f076343d8ad826195b9d0789c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 } } peers { permanent_uuid: "4608c5fbd2a240bebffbe55bbc8b1acd" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34099 } } peers { permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 } } }
I20260809 15:01:13.145710 31077 tablet_replica.cc:442] T ca090fc4b6024705b14a24c38d41c116 P 4608c5fbd2a240bebffbe55bbc8b1acd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:13.152662 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.146200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.27.186.130:34099)
0809 15:01:13.146385 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.146392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.146693 (+ 301us) server_negotiation.cc:407] Connection header received
0809 15:01:13.147005 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.147010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.147126 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.147265 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:13.147971 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.148941 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.150427 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.150846 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.150957 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.151377 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.152025 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.152158 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.152294 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.152385 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 15:01:13.170251 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.164307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:50079 (local address 127.27.186.131:37027)
0809 15:01:13.164492 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.164499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.164856 (+ 357us) server_negotiation.cc:407] Connection header received
0809 15:01:13.164992 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.164997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.165087 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.165176 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:13.165513 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.166690 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.168086 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.168886 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.169026 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.169433 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.169585 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.169677 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.169824 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.169901 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:01:13.170265 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.164229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:37027 (local address 127.27.186.130:50079)
0809 15:01:13.164766 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:13.164781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:13.164884 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:13.165194 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:13.165198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:13.165207 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:13.165382 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:13.165389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.166830 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.166835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:13.167953 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:13.167966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.169063 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.169071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:13.169202 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.169257 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.169687 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.169697 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 15:01:13.169792 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 15:01:13.169917 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":271,"threads_started":1}
I20260809 15:01:13.181470 30942 raft_consensus.cc:1261] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Refusing update from remote peer 4608c5fbd2a240bebffbe55bbc8b1acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:13.181547 30459 raft_consensus.cc:1261] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Refusing update from remote peer 4608c5fbd2a240bebffbe55bbc8b1acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:13.182089 31077 consensus_queue.cc:1035] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2cc87f076343d8ad826195b9d0789c" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:13.182703 31116 consensus_queue.cc:1035] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca015c5195454c989d61cbe384d3f585" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:13.190171 31149 mvcc.cc:205] Tried to move back new op lower bound from 7316634309344780288 to 7316634307726766080. Current Snapshot: MvccSnapshot[applied={T|T < 7316634309344780288}]
I20260809 15:01:13.255312 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.234405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34099 (local address 127.27.186.131:36081)
0809 15:01:13.245955 (+ 11550us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:13.245972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:13.246141 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:13.246844 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:13.246848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:13.246860 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:13.247222 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:13.247232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.248650 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.248655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:13.252563 (+ 3908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:13.252584 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.253766 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.253772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:13.253960 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.254046 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.254533 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.254558 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 15:01:13.254644 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 15:01:13.254788 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11370,"thread_start_us":11189,"threads_started":1}
I20260809 15:01:13.255920 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.234875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:36081 (local address 127.27.186.130:34099)
0809 15:01:13.235051 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.235058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.246522 (+ 11464us) server_negotiation.cc:407] Connection header received
0809 15:01:13.246619 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.246624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.246725 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.246828 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:13.247396 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.248489 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.252752 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.253432 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.253522 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.254253 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:13.254387 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:13.254477 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.255540 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.255638 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 15:01:13.309398 30198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:13.309661 30198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:13.313476 30198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42a8152054a1466696d4a36af97a931f: Sending DeleteTablet for 3 replicas of tablet 39845f1af4974d37a14858af36b068aa
I20260809 15:01:13.314754 30424 tablet_service.cc:1514] Processing DeleteTablet for tablet 39845f1af4974d37a14858af36b068aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:13 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34372
I20260809 15:01:13.315212 30666 tablet_service.cc:1514] Processing DeleteTablet for tablet 39845f1af4974d37a14858af36b068aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:13 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:58854
I20260809 15:01:13.317646 30896 tablet_service.cc:1514] Processing DeleteTablet for tablet 39845f1af4974d37a14858af36b068aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:13 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:40678
I20260809 15:01:13.318413 31181 tablet_replica.cc:330] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: stopping tablet replica
I20260809 15:01:13.326218 31181 raft_consensus.cc:2229] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:13.326788 31187 tablet_replica.cc:330] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: stopping tablet replica
I20260809 15:01:13.327081 31181 raft_consensus.cc:2258] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:13.327351 31187 raft_consensus.cc:2229] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:13.328624 31181 ts_tablet_manager.cc:1905] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:13.329279 31187 raft_consensus.cc:2258] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:13.330451 31187 ts_tablet_manager.cc:1905] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:13.334749 31192 tablet_replica.cc:330] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: stopping tablet replica
I20260809 15:01:13.339150 31192 raft_consensus.cc:2229] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:13.339542 31192 raft_consensus.cc:2258] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:13.341427 31192 ts_tablet_manager.cc:1905] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 15:01:13 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:13.360909 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.340077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.27.186.129:45607)
0809 15:01:13.340248 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.340253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.341225 (+ 972us) server_negotiation.cc:407] Connection header received
0809 15:01:13.341498 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.341504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.341598 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.341673 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:13.344045 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.345108 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.347166 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.347849 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.347953 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.353583 (+ 5630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:13.353597 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:13.353614 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:13.353651 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:13.357411 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.358320 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.358330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.358333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.358494 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.358931 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.358936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.358940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.359359 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:13.359673 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.360275 (+ 602us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.360517 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 15:01:13.362562 30539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34380
I20260809 15:01:13.396150 31181 ts_tablet_manager.cc:1918] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:13.396323 31181 log.cc:1196] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/39845f1af4974d37a14858af36b068aa
I20260809 15:01:13.396817 31181 ts_tablet_manager.cc:1939] T 39845f1af4974d37a14858af36b068aa P ca015c5195454c989d61cbe384d3f585: Deleting consensus metadata
I20260809 15:01:13.397359 31192 ts_tablet_manager.cc:1918] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:13.397522 31192 log.cc:1196] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/39845f1af4974d37a14858af36b068aa
I20260809 15:01:13.398031 31192 ts_tablet_manager.cc:1939] T 39845f1af4974d37a14858af36b068aa P 8c2cc87f076343d8ad826195b9d0789c: Deleting consensus metadata
I20260809 15:01:13.398538 31187 ts_tablet_manager.cc:1918] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:13.398705 31187 log.cc:1196] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/39845f1af4974d37a14858af36b068aa
I20260809 15:01:13.399204 31187 ts_tablet_manager.cc:1939] T 39845f1af4974d37a14858af36b068aa P 4608c5fbd2a240bebffbe55bbc8b1acd: Deleting consensus metadata
I20260809 15:01:13.406668 30171 catalog_manager.cc:4841] TS ca015c5195454c989d61cbe384d3f585 (127.27.186.129:45607): tablet 39845f1af4974d37a14858af36b068aa (table test-table [id=065c861133fc49b3bdb22af8541e7ee8]) successfully deleted
I20260809 15:01:13.413419 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.376449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.27.186.190:37209)
0809 15:01:13.394367 (+ 17918us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.394374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.394394 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:13.394458 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.394463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.394550 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.394656 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:13.397324 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.398545 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.400921 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.401660 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.402932 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.405646 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:13.405653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:13.405658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:13.405699 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:13.409214 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.410106 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.410112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.410117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.410271 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.410765 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.410771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.410774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.411154 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:13.411401 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.412756 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.412990 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17790,"thread_start_us":19869,"threads_started":1}
W20260809 15:01:13.414794 30198 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062
I20260809 15:01:13.419181 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30322
I20260809 15:01:13.423553 30172 catalog_manager.cc:4841] TS 8c2cc87f076343d8ad826195b9d0789c (127.27.186.131:37027): tablet 39845f1af4974d37a14858af36b068aa (table test-table [id=065c861133fc49b3bdb22af8541e7ee8]) successfully deleted
I20260809 15:01:13.447199 30168 catalog_manager.cc:4841] TS 4608c5fbd2a240bebffbe55bbc8b1acd (127.27.186.130:34099): tablet 39845f1af4974d37a14858af36b068aa (table test-table [id=065c861133fc49b3bdb22af8541e7ee8]) successfully deleted
I20260809 15:01:13.450273 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.421362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.27.186.129:45607)
0809 15:01:13.421543 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.421550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.432529 (+ 10979us) server_negotiation.cc:407] Connection header received
0809 15:01:13.432672 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.432678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.432780 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.432889 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:13.434775 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.435876 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.437698 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.438388 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.438506 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.441706 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:13.441714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:13.441719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:13.441763 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:13.445739 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.446599 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.446605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.446608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.446772 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.447896 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.447902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.447905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.448277 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:13.448601 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.449466 (+ 865us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.449762 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 15:01:13.451924 30539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34384
W20260809 15:01:13.452665 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:13.452836 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30322
I20260809 15:01:13.496953 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30555
Aug 09 15:01:13 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:13.519842 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.498703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.27.186.130:34099)
0809 15:01:13.498900 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.498906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.498926 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:13.498987 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.498992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.499064 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.499190 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:13.500881 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.502108 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.504011 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.504759 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.504896 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.510764 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:13.510788 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:13.510806 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:13.510857 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:13.515267 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.516198 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.516208 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.516212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.516400 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.516931 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.516936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.516940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.517443 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:13.517794 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.519155 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.519407 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 15:01:13.520931 30773 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880
W20260809 15:01:13.521809 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30555: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:13.522245 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30555
I20260809 15:01:13.548473 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30789
Aug 09 15:01:13 dist-test-slave-q9h9 krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:13.565164 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:13.549218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.27.186.131:37027)
0809 15:01:13.549435 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:13.549447 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:13.549706 (+ 259us) server_negotiation.cc:407] Connection header received
0809 15:01:13.549946 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:13.549953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:13.550062 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:13.550170 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:13.551572 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.552379 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.553708 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:13.554417 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:13.554540 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:13.558781 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:13.558794 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:13.558808 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:13.558843 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:13.562289 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.562950 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.562957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.562960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.563113 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:13.563503 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:13.563507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:13.563510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:13.563924 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:13.564279 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:13.564513 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 15:01:13.564746 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 15:01:13.565876 31019 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40692
W20260809 15:01:13.566566 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30789: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:13.566709 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30789
I20260809 15:01:13.591207 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30144
I20260809 15:01:13.592219 30264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:13.710896 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30144
2026-08-09T15:01:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4856 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:14 dist-test-slave-q9h9 krb5kdc[31206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:14 dist-test-slave-q9h9 krb5kdc[31206](info): set up 2 sockets
Aug 09 15:01:14 dist-test-slave-q9h9 krb5kdc[31206](info): commencing operation
krb5kdc: starting...
W20260809 15:01:15.566262 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.813s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:15 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:15.731033 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:46089
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:44093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:16.034379 31228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:16.034614 31228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:16.034695 31228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:16.041448 31228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:16.041531 31228 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 15:01:16.041580 31228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:16.041636 31228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:16.041683 31228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:16.041740 31228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:16.050823 31228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:44093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:46089
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:16.051815 31228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:16.052996 31228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:16.063599 31243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:16.063637 31239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:16.063676 31240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:16.064584 31228 server_base.cc:1034] running on GCE node
I20260809 15:01:16.065201 31228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:16.066413 31228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:16.067639 31228 hybrid_clock.cc:648] HybridClock initialized: now 1786287676067576 us; error 33 us; skew 500 ppm
I20260809 15:01:16.067925 31228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:16 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:16.070788 31228 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:16.072141 31228 webserver.cc:460] Webserver started at http://127.27.186.190:38117/ using document root <none> and password file <none>
I20260809 15:01:16.072737 31228 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:16.072822 31228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:16.073067 31228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:16.075847 31228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "b1051ec972e84c29967ed791d0643c11"
format_stamp: "Formatted at 2026-08-09 15:01:16 on dist-test-slave-q9h9"
server_key: "e1e7e89e06a241ff44461bab37e53d9c"
server_key_iv: "219e5028b14f2ca7d1be8e5253fd013b"
server_key_version: "encryptionkey@0"
I20260809 15:01:16.076617 31228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "b1051ec972e84c29967ed791d0643c11"
format_stamp: "Formatted at 2026-08-09 15:01:16 on dist-test-slave-q9h9"
server_key: "e1e7e89e06a241ff44461bab37e53d9c"
server_key_iv: "219e5028b14f2ca7d1be8e5253fd013b"
server_key_version: "encryptionkey@0"
I20260809 15:01:16.081264 31228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 15:01:16.084450 31257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:16.085636 31228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 15:01:16.085765 31228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "b1051ec972e84c29967ed791d0643c11"
format_stamp: "Formatted at 2026-08-09 15:01:16 on dist-test-slave-q9h9"
server_key: "e1e7e89e06a241ff44461bab37e53d9c"
server_key_iv: "219e5028b14f2ca7d1be8e5253fd013b"
server_key_version: "encryptionkey@0"
I20260809 15:01:16.085928 31228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:16.102818 31228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:16.106422 31228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:16.106621 31228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:16.125787 31228 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:46089
I20260809 15:01:16.125841 31352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:46089 every 8 connection(s)
I20260809 15:01:16.126968 31228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:16.130307 31353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:16.134564 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31228
I20260809 15:01:16.134775 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:16.135600 28394 external_mini_cluster.cc:1428] Setting key cbcdc2b42c886bd56e6c31811dcf17b6
I20260809 15:01:16.141217 31353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Bootstrap starting.
I20260809 15:01:16.144596 31353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:16.145710 31353 log.cc:824] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:16.148828 31353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: No bootstrap required, opened a new log
Aug 09 15:01:16 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:16.156277 31353 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1051ec972e84c29967ed791d0643c11" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46089 } }
I20260809 15:01:16.156598 31353 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:16.156716 31353 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1051ec972e84c29967ed791d0643c11, State: Initialized, Role: FOLLOWER
I20260809 15:01:16.157312 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:16.137532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.27.186.190:46089)
0809 15:01:16.138304 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:16.138314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:16.138357 (+ 43us) server_negotiation.cc:407] Connection header received
0809 15:01:16.140013 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:16.140033 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:16.140361 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:16.140633 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:16.142350 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.143587 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.145244 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.145925 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.146089 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:16.150416 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:16.150424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:16.150436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:16.150470 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:16.153484 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:16.154134 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:16.154143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:16.154151 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:16.154326 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:16.154692 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:16.154695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:16.154697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:16.155209 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:16.155497 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:16.155933 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 15:01:16.156273 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":353,"threads_started":1}
I20260809 15:01:16.157269 31353 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1051ec972e84c29967ed791d0643c11" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46089 } }
I20260809 15:01:16.157502 31353 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:16.157635 31353 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:16.157801 31353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:16.160010 31353 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1051ec972e84c29967ed791d0643c11" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46089 } }
I20260809 15:01:16.160471 31353 leader_election.cc:302] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1051ec972e84c29967ed791d0643c11; no voters:
I20260809 15:01:16.160986 31353 leader_election.cc:288] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:16.161098 31360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:16.162320 31360 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [term 1 LEADER]: Becoming Leader. State: Replica: b1051ec972e84c29967ed791d0643c11, State: Running, Role: LEADER
I20260809 15:01:16.162798 31360 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1051ec972e84c29967ed791d0643c11" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46089 } }
I20260809 15:01:16.162955 31353 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:16.165520 31362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1051ec972e84c29967ed791d0643c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1051ec972e84c29967ed791d0643c11" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46089 } } }
I20260809 15:01:16.165951 31362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:16.165508 31364 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1051ec972e84c29967ed791d0643c11. Latest consensus state: current_term: 1 leader_uuid: "b1051ec972e84c29967ed791d0643c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1051ec972e84c29967ed791d0643c11" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46089 } } }
I20260809 15:01:16.166329 31364 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:16.170668 31378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:16.174558 31378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:16.182268 31378 catalog_manager.cc:1347] Generated new cluster ID: d3abce561b364b32b87b9b34cf5a9009
I20260809 15:01:16.182363 31378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:16.202567 31378 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:16.203873 31378 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:16.213809 31378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Generated new TSK 0
I20260809 15:01:16.214524 31378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:16.263242 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:46089
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:44093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:16.669713 31398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:16.669946 31398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:16.670114 31398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:16.676951 31398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:16.677039 31398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:16.677150 31398 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 15:01:16.677207 31398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:01:16.677278 31398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:16.686309 31398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:44093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:46089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:16.687330 31398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:16.688491 31398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:16.700253 31404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:16.700330 31407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:16.700374 31409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:16.701370 31398 server_base.cc:1034] running on GCE node
I20260809 15:01:16.701905 31398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:16.702756 31398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:16.703948 31398 hybrid_clock.cc:648] HybridClock initialized: now 1786287676703899 us; error 42 us; skew 500 ppm
I20260809 15:01:16.704221 31398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:16 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:16.711499 31398 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:16.712823 31398 webserver.cc:460] Webserver started at http://127.27.186.129:46111/ using document root <none> and password file <none>
I20260809 15:01:16.713416 31398 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:16.713531 31398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:16.713810 31398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:16.716526 31398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "103c4aaf55d84e9d9640e1afc016da0b"
format_stamp: "Formatted at 2026-08-09 15:01:16 on dist-test-slave-q9h9"
server_key: "9c29c1437dd76d0b401201b2fe919a09"
server_key_iv: "eea46a0bbf4d82de43c17b375ff290ad"
server_key_version: "encryptionkey@0"
I20260809 15:01:16.717216 31398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "103c4aaf55d84e9d9640e1afc016da0b"
format_stamp: "Formatted at 2026-08-09 15:01:16 on dist-test-slave-q9h9"
server_key: "9c29c1437dd76d0b401201b2fe919a09"
server_key_iv: "eea46a0bbf4d82de43c17b375ff290ad"
server_key_version: "encryptionkey@0"
I20260809 15:01:16.722119 31398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 15:01:16.725256 31421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:16.726478 31398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:01:16.726619 31398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "103c4aaf55d84e9d9640e1afc016da0b"
format_stamp: "Formatted at 2026-08-09 15:01:16 on dist-test-slave-q9h9"
server_key: "9c29c1437dd76d0b401201b2fe919a09"
server_key_iv: "eea46a0bbf4d82de43c17b375ff290ad"
server_key_version: "encryptionkey@0"
I20260809 15:01:16.726783 31398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:16.738723 31398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:16.742478 31398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:16.742727 31398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:16.745191 31398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:16.745324 31398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:16.745499 31398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:16.745590 31398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:01:16 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:16.766193 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:16.745608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:59025 (local address 127.27.186.190:46089)
0809 15:01:16.746101 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:16.746107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:16.747105 (+ 998us) server_negotiation.cc:407] Connection header received
0809 15:01:16.749053 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:16.749059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:16.749149 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:16.749244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:16.751077 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.752070 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.754976 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.755432 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.755562 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:16.759585 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:16.759591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:16.759596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:16.759637 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:16.762763 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:16.763800 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:16.763806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:16.763809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:16.763962 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:16.764468 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:16.764473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:16.764475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:16.764768 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:16.764901 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:16.765573 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 15:01:16.765805 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260809 15:01:16.767059 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:16.746220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.27.186.129:59025)
0809 15:01:16.746959 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:16.746999 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:16.748797 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:16.749478 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:16.749485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:16.750064 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:16.750896 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:16.750918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.752245 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.752251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:16.754779 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:16.754804 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.755666 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.755674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:16.755864 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:16.756463 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:16.756489 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:16.759352 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:16.762950 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:16.762962 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:16.762979 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:16.763649 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:16.764084 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:16.764089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:16.764093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:16.764315 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:16.764906 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:16.764914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:16.765219 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 15:01:16.765434 (+ 215us) client_negotiation.cc:240] Negotiation successful
0809 15:01:16.765877 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":312,"threads_started":1}
I20260809 15:01:16.788956 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:16.776405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.27.186.129:45839)
0809 15:01:16.776962 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:16.776978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:16.777091 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:16.777433 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:16.777441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:16.778042 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:16.778543 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:16.778559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.779750 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.779754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:16.781207 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:16.781219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.781935 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.781942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:16.782155 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:16.782860 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:16.782879 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:16.783981 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:16.786574 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:16.786579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:16.786587 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:16.786983 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:16.787332 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:16.787341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:16.787345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:16.787507 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:16.788120 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:16.788123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:16.788274 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 15:01:16.788379 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 15:01:16.788649 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":328,"threads_started":1}
I20260809 15:01:16.789127 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:16.776310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:45839 (local address 127.27.186.190:46089)
0809 15:01:16.776458 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:16.776464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:16.777058 (+ 594us) server_negotiation.cc:407] Connection header received
0809 15:01:16.777204 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:16.777209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:16.777284 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:16.777379 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:16.778708 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.779577 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.781349 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:16.781760 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:16.781866 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:16.784165 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:16.784171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:16.784174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:16.784203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:16.786452 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:16.787091 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:16.787097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:16.787100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:16.787219 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:16.787634 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:16.787641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:16.787645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:16.787994 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:16.788161 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:16.788518 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 15:01:16.788757 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:01:16.798730 31398 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:44207
I20260809 15:01:16.798804 31620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:44207 every 8 connection(s)
I20260809 15:01:16.799813 31398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:16.801492 31623 heartbeater.cc:347] Connected to a master server at 127.27.186.190:46089
I20260809 15:01:16.801699 31623 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:16.802325 31623 heartbeater.cc:511] Master 127.27.186.190:46089 requested a full tablet report, sending...
I20260809 15:01:16.803316 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31398
I20260809 15:01:16.803499 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:16.804136 28394 external_mini_cluster.cc:1428] Setting key b603eb6957fd47216a382b98d4bbb023
I20260809 15:01:16.804207 31288 ts_manager.cc:194] Registered new tserver with Master: 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207)
I20260809 15:01:16.806056 31288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:59025
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:16.842748 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:46089
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:44093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:17.146296 31630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:17.146554 31630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:17.146697 31630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:17.153610 31630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:17.153714 31630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:17.153836 31630 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 15:01:17.153903 31630 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:01:17.154008 31630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:17.163672 31630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:44093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:46089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:17.164814 31630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:17.165966 31630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:17.177914 31644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:17.178054 31648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:17.178002 31641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:17.178550 31630 server_base.cc:1034] running on GCE node
I20260809 15:01:17.179068 31630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:17.179862 31630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:17.181051 31630 hybrid_clock.cc:648] HybridClock initialized: now 1786287677181004 us; error 35 us; skew 500 ppm
I20260809 15:01:17.181332 31630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:17 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:17.184206 31630 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:17.185518 31630 webserver.cc:460] Webserver started at http://127.27.186.130:35817/ using document root <none> and password file <none>
I20260809 15:01:17.186187 31630 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:17.186309 31630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:17.186583 31630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:17.189229 31630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
format_stamp: "Formatted at 2026-08-09 15:01:17 on dist-test-slave-q9h9"
server_key: "cc252b90c62fc58eac6acd58cc80abd1"
server_key_iv: "d549f1d6713b6351b35b4c76bbcaa5aa"
server_key_version: "encryptionkey@0"
I20260809 15:01:17.189939 31630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
format_stamp: "Formatted at 2026-08-09 15:01:17 on dist-test-slave-q9h9"
server_key: "cc252b90c62fc58eac6acd58cc80abd1"
server_key_iv: "d549f1d6713b6351b35b4c76bbcaa5aa"
server_key_version: "encryptionkey@0"
I20260809 15:01:17.194705 31630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 15:01:17.197815 31662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:17.198967 31630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:01:17.199106 31630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
format_stamp: "Formatted at 2026-08-09 15:01:17 on dist-test-slave-q9h9"
server_key: "cc252b90c62fc58eac6acd58cc80abd1"
server_key_iv: "d549f1d6713b6351b35b4c76bbcaa5aa"
server_key_version: "encryptionkey@0"
I20260809 15:01:17.199275 31630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:17.221155 31630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:17.224756 31630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:17.225000 31630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:17.227648 31630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:17.227778 31630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:17.227897 31630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:17.228061 31630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 09 15:01:17 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:17.248479 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.228112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:48685 (local address 127.27.186.190:46089)
0809 15:01:17.228339 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.228345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.229914 (+ 1569us) server_negotiation.cc:407] Connection header received
0809 15:01:17.232262 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.232267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.232358 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.232442 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:17.233855 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.234864 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.237101 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.237772 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.237924 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.242001 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:17.242008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:17.242033 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:17.242073 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:17.245335 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.246266 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.246272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.246275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.246409 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.246819 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.246824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.246828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.247099 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:17.247244 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.247892 (+ 648us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.248102 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 15:01:17.249820 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.228848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.27.186.130:48685)
0809 15:01:17.229738 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.229778 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.231993 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.232678 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.232686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.233165 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:17.233688 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.233704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.235018 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.235022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.236932 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.236951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.238054 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.238063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.238376 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.239118 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:17.239144 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:17.241796 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:17.245460 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.245469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.245482 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.246153 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.246513 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.246518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.246522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.246705 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.247219 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:17.247226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:17.247542 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.247772 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.248233 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":387,"threads_started":1}
I20260809 15:01:17.269923 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.255633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:57781 (local address 127.27.186.190:46089)
0809 15:01:17.255890 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.255896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.256450 (+ 554us) server_negotiation.cc:407] Connection header received
0809 15:01:17.256613 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.256619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.256689 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.256770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:17.257771 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.258639 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.260632 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.261124 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.261248 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.263920 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:17.263927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:17.263932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:17.263969 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:17.266778 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.267722 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.267728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.267732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.267877 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.268260 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.268266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.268270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.268596 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:17.268732 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.269281 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.269481 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260809 15:01:17.269980 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.255715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.27.186.130:57781)
0809 15:01:17.256344 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.256361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.256494 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.256823 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.256829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.257320 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:17.257624 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.257634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.258829 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.258836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.260396 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.260414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.261571 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.261585 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.261742 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.262451 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:17.262479 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:17.263740 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:17.266948 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.266957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.266962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.267598 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.268000 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.268011 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.268016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.268150 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.268795 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:17.268801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:17.268974 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.269159 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.269481 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":375,"threads_started":1}
I20260809 15:01:17.280472 31630 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33527
I20260809 15:01:17.280535 31878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33527 every 8 connection(s)
I20260809 15:01:17.281684 31630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:17.283620 31879 heartbeater.cc:347] Connected to a master server at 127.27.186.190:46089
I20260809 15:01:17.283831 31879 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:17.284425 31879 heartbeater.cc:511] Master 127.27.186.190:46089 requested a full tablet report, sending...
I20260809 15:01:17.285640 31288 ts_manager.cc:194] Registered new tserver with Master: d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:17.286520 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31630
I20260809 15:01:17.286660 31288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:48685
I20260809 15:01:17.286713 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:17.287257 28394 external_mini_cluster.cc:1428] Setting key e60f01baec05efa48640e772e6aa81fb
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:17.324563 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:46089
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:44093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:17.631729 31889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:17.632055 31889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:17.632226 31889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:17.639873 31889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:17.639969 31889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:17.640100 31889 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 15:01:17.640163 31889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:01:17.640271 31889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:17.649312 31889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:44093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:46089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:17.650404 31889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:17.651575 31889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:17.663470 31896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:17.663503 31897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:17.663523 31899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:17.664541 31889 server_base.cc:1034] running on GCE node
I20260809 15:01:17.665038 31889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:17.665823 31889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:17.667006 31889 hybrid_clock.cc:648] HybridClock initialized: now 1786287677666965 us; error 33 us; skew 500 ppm
I20260809 15:01:17.667286 31889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:17 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:17.670230 31889 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:17.671494 31889 webserver.cc:460] Webserver started at http://127.27.186.131:40529/ using document root <none> and password file <none>
I20260809 15:01:17.672071 31889 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:17.672185 31889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:17.672472 31889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:17.675109 31889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "27758f421417413b94fc31bacf4c3255"
format_stamp: "Formatted at 2026-08-09 15:01:17 on dist-test-slave-q9h9"
server_key: "6dd23dbeacd1475edf64000c9e6853da"
server_key_iv: "bdb1739417d8e455912fb0768d150f91"
server_key_version: "encryptionkey@0"
I20260809 15:01:17.675804 31889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "27758f421417413b94fc31bacf4c3255"
format_stamp: "Formatted at 2026-08-09 15:01:17 on dist-test-slave-q9h9"
server_key: "6dd23dbeacd1475edf64000c9e6853da"
server_key_iv: "bdb1739417d8e455912fb0768d150f91"
server_key_version: "encryptionkey@0"
I20260809 15:01:17.680526 31889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:17.683688 31910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:17.684865 31889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:17.685007 31889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "27758f421417413b94fc31bacf4c3255"
format_stamp: "Formatted at 2026-08-09 15:01:17 on dist-test-slave-q9h9"
server_key: "6dd23dbeacd1475edf64000c9e6853da"
server_key_iv: "bdb1739417d8e455912fb0768d150f91"
server_key_version: "encryptionkey@0"
I20260809 15:01:17.685173 31889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:17.699210 31889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:17.702745 31889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:17.702989 31889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:17.705369 31889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:17.705483 31889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:17.705593 31889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:17.705673 31889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:01:17 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:17.727917 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.706634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.27.186.131:38703)
0809 15:01:17.707481 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.707523 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.709377 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.710131 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.710142 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.710812 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:17.711587 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.711616 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.712942 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.712947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.715078 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.715095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.715908 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.715917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.716122 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.716790 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:17.716818 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:17.719666 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:17.723344 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.723353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.723366 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.723956 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.724416 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.724421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.724428 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.724708 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.725401 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:17.725406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:17.725793 (+ 387us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.726142 (+ 349us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.726611 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":382,"threads_started":1}
I20260809 15:01:17.728785 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.705935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:38703 (local address 127.27.186.190:46089)
0809 15:01:17.706144 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.706149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.707627 (+ 1478us) server_negotiation.cc:407] Connection header received
0809 15:01:17.709662 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.709668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.709753 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.709882 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:17.711781 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.712783 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.715245 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.715662 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.715777 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.719929 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:17.719936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:17.719940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:17.720015 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:17.723214 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.724095 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.724101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.724105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.724287 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.724828 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.724834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.724838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.725183 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:17.725329 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.728266 (+ 2937us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.728439 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 15:01:17.741488 31889 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:39711
I20260809 15:01:17.741555 32129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:39711 every 8 connection(s)
I20260809 15:01:17.742812 31889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:17.744733 32132 heartbeater.cc:347] Connected to a master server at 127.27.186.190:46089
I20260809 15:01:17.744977 32132 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:17.745784 32132 heartbeater.cc:511] Master 127.27.186.190:46089 requested a full tablet report, sending...
I20260809 15:01:17.747277 31288 ts_manager.cc:194] Registered new tserver with Master: 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711)
I20260809 15:01:17.747850 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31889
I20260809 15:01:17.748027 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:17.748299 31288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:38703
I20260809 15:01:17.748539 28394 external_mini_cluster.cc:1428] Setting key 47f8179486fb6d74f54e2a26b44279f0
I20260809 15:01:17.751667 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:01:17.754161 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.741730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.27.186.131:59537)
0809 15:01:17.742393 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.742407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.742524 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.742830 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.742834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.743244 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:17.743595 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.743610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.744918 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.744923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.746466 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.746480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.747288 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.747293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.747435 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.747906 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:17.747924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:17.748627 (+ 703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:17.752018 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.752021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.752024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.752368 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.752791 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:17.752794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:17.752796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:17.752906 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:17.753384 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:17.753387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:17.753516 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.753650 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.753846 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":361,"threads_started":1}
I20260809 15:01:17.754459 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.741596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:59537 (local address 127.27.186.190:46089)
0809 15:01:17.741805 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.741811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.742485 (+ 674us) server_negotiation.cc:407] Connection header received
0809 15:01:17.742631 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.742637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.742709 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.742785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:17.743786 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.744733 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.746585 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.747118 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.747247 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.748829 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:17.748834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:17.748839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:17.748866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:17.751908 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.752527 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.752533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.752537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.752686 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.752988 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.752992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.752996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.753296 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:17.753428 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.753761 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.753971 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Aug 09 15:01:17 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 15:01:17 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:17.796690 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.784577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.27.186.190:46089)
0809 15:01:17.784768 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.784783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.785014 (+ 231us) server_negotiation.cc:407] Connection header received
0809 15:01:17.785155 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.785158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.785215 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.785284 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:17.786589 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.787262 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.788479 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.788961 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.789050 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.792519 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:17.792524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:17.792527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:17.792554 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:17.794709 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.795246 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.795250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.795252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.795351 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:17.795620 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:17.795623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:17.795625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:17.795862 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:17.796021 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.796223 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.796365 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:01:17.800284 31288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:17.807148 31288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:17.808961 31623 heartbeater.cc:503] Master 127.27.186.190:46089 was elected leader, sending a full tablet report...
I20260809 15:01:17.828970 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.818920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.27.186.129:44207)
0809 15:01:17.821089 (+ 2169us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.821095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.821598 (+ 503us) server_negotiation.cc:407] Connection header received
0809 15:01:17.821888 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.821895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.822117 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.822258 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:17.823058 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.824240 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.825971 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.827912 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.828078 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.828237 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.828532 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.828690 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052,"thread_start_us":1939,"threads_started":1}
I20260809 15:01:17.829018 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.818398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:44207 (local address 127.0.0.1:37592)
0809 15:01:17.821492 (+ 3094us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.821513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.821698 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.822481 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.822485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.822509 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:17.822883 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.822893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.824380 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.824383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.825826 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.825838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.828128 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.828134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.828321 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.828372 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.828479 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.828641 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"mutex_wait_us":1,"thread_start_us":575,"threads_started":1}
I20260809 15:01:17.831092 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.818081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39711 (local address 127.0.0.1:47510)
0809 15:01:17.819396 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.819435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.819596 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.820678 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.820685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.820718 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:17.821037 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.821052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.826318 (+ 5266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.826327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.828156 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.828169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.830250 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.830256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.830448 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.830480 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.830616 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.830746 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":3141,"threads_started":1}
I20260809 15:01:17.831868 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.818368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.27.186.131:39711)
0809 15:01:17.820213 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.820221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.820252 (+ 31us) server_negotiation.cc:407] Connection header received
0809 15:01:17.820329 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.820335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.820486 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.820627 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:17.821295 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.822898 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.828366 (+ 5468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.829747 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.829933 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.830119 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.830680 (+ 561us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.831562 (+ 882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1700,"thread_start_us":1520,"threads_started":1}
I20260809 15:01:17.832726 31501 tablet_service.cc:1467] Processing CreateTablet for tablet e7ef2fe052114dc993010e9ae13d3ca2 (DEFAULT_TABLE table=test-table [id=8ee6f3419d074e02b2a7c80b49e3ba61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:17.834177 31501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ef2fe052114dc993010e9ae13d3ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:17.834412 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.821576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33527 (local address 127.0.0.1:59114)
0809 15:01:17.826338 (+ 4762us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.826354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.826469 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.827489 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.827493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.827517 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:17.827954 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.827966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.829931 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.829936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.831985 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.832001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.833656 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.833663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.833832 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.833863 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.833957 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.834105 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4648,"thread_start_us":4523,"threads_started":1}
I20260809 15:01:17.834479 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.821817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.27.186.130:33527)
0809 15:01:17.826958 (+ 5141us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.826965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.826990 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:17.827078 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.827084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.827288 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.828164 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:17.828365 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.829744 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.832139 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.833430 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.833609 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.833759 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.834012 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.834210 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4980,"thread_start_us":4841,"threads_started":1}
I20260809 15:01:17.835201 32003 tablet_service.cc:1467] Processing CreateTablet for tablet e7ef2fe052114dc993010e9ae13d3ca2 (DEFAULT_TABLE table=test-table [id=8ee6f3419d074e02b2a7c80b49e3ba61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:17.836622 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ef2fe052114dc993010e9ae13d3ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:17.837966 31756 tablet_service.cc:1467] Processing CreateTablet for tablet e7ef2fe052114dc993010e9ae13d3ca2 (DEFAULT_TABLE table=test-table [id=8ee6f3419d074e02b2a7c80b49e3ba61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:17.839284 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ef2fe052114dc993010e9ae13d3ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:17.846606 32160 tablet_bootstrap.cc:492] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Bootstrap starting.
I20260809 15:01:17.848316 32161 tablet_bootstrap.cc:492] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Bootstrap starting.
I20260809 15:01:17.850512 32160 tablet_bootstrap.cc:654] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:17.851670 32161 tablet_bootstrap.cc:654] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:17.851795 32160 log.cc:824] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:17.852113 32162 tablet_bootstrap.cc:492] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Bootstrap starting.
I20260809 15:01:17.852982 32161 log.cc:824] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:17.854988 32160 tablet_bootstrap.cc:492] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: No bootstrap required, opened a new log
I20260809 15:01:17.855214 32160 ts_tablet_manager.cc:1397] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 15:01:17.855343 32162 tablet_bootstrap.cc:654] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:17.855759 32161 tablet_bootstrap.cc:492] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: No bootstrap required, opened a new log
I20260809 15:01:17.856045 32161 ts_tablet_manager.cc:1397] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:01:17.856544 32162 log.cc:824] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:17.860455 32162 tablet_bootstrap.cc:492] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: No bootstrap required, opened a new log
I20260809 15:01:17.860715 32162 ts_tablet_manager.cc:1397] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260809 15:01:17.862987 32160 raft_consensus.cc:348] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.863309 32160 raft_consensus.cc:374] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:17.863406 32160 raft_consensus.cc:729] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Initialized, Role: FOLLOWER
I20260809 15:01:17.863945 32160 consensus_queue.cc:260] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.864946 32132 heartbeater.cc:503] Master 127.27.186.190:46089 was elected leader, sending a full tablet report...
I20260809 15:01:17.865576 32160 ts_tablet_manager.cc:1428] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260809 15:01:17.865451 32161 raft_consensus.cc:348] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.865815 32161 raft_consensus.cc:374] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:17.865898 32161 raft_consensus.cc:729] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Initialized, Role: FOLLOWER
I20260809 15:01:17.866510 32161 consensus_queue.cc:260] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.868186 32161 ts_tablet_manager.cc:1428] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260809 15:01:17.871086 32162 raft_consensus.cc:348] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.871377 32162 raft_consensus.cc:374] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:17.871495 32162 raft_consensus.cc:729] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42a4d8c2de64aaeaf1ef7c0381e8efe, State: Initialized, Role: FOLLOWER
I20260809 15:01:17.871961 32162 consensus_queue.cc:260] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.872776 31879 heartbeater.cc:503] Master 127.27.186.190:46089 was elected leader, sending a full tablet report...
I20260809 15:01:17.873313 32162 ts_tablet_manager.cc:1428] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 15:01:17.930912 32170 raft_consensus.cc:482] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:17.931159 32170 raft_consensus.cc:504] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.932618 32170 leader_election.cc:288] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:17.939355 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.932972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:42347 (local address 127.27.186.131:39711)
0809 15:01:17.933132 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.933136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.933622 (+ 486us) server_negotiation.cc:407] Connection header received
0809 15:01:17.933807 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.933811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.933884 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.933976 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:17.934715 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.935571 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.937487 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.938302 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.938444 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.938514 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.938990 (+ 476us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.939111 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:01:17.939378 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.932917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39711 (local address 127.27.186.129:42347)
0809 15:01:17.933523 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.933542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.933671 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.934158 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.934162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.934183 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:17.934576 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.934590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.935697 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.935711 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.937333 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.937346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.938497 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.938504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.938823 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.938851 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.938950 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.939076 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":2,"spinlock_wait_cycles":35072,"thread_start_us":307,"threads_started":1}
I20260809 15:01:17.939407 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.933818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:55921 (local address 127.27.186.130:33527)
0809 15:01:17.933990 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:17.933995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:17.934319 (+ 324us) server_negotiation.cc:407] Connection header received
0809 15:01:17.934469 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:17.934473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:17.934542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:17.934638 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:17.935174 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.935948 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.937598 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.938438 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.938582 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.938647 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:17.939064 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 15:01:17.939178 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:01:17.940019 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:17.933621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33527 (local address 127.27.186.129:55921)
0809 15:01:17.934219 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:17.934235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:17.934355 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:17.934649 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:17.934653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:17.934676 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:17.935041 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.935053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.936078 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.936082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.937449 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:17.937465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:17.938656 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:17.938662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:17.938880 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:17.938906 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 15:01:17.939547 (+ 641us) client_negotiation.cc:240] Negotiation successful
0809 15:01:17.939666 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":48,"spinlock_wait_cycles":2432,"thread_start_us":306,"threads_started":1}
I20260809 15:01:17.940531 32040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ef2fe052114dc993010e9ae13d3ca2" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27758f421417413b94fc31bacf4c3255" is_pre_election: true
I20260809 15:01:17.940963 32040 raft_consensus.cc:2454] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 0.
I20260809 15:01:17.940918 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ef2fe052114dc993010e9ae13d3ca2" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" is_pre_election: true
I20260809 15:01:17.941313 31801 raft_consensus.cc:2454] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 0.
I20260809 15:01:17.941566 31422 leader_election.cc:302] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4aaf55d84e9d9640e1afc016da0b, 27758f421417413b94fc31bacf4c3255; no voters:
I20260809 15:01:17.941924 32170 raft_consensus.cc:2781] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:17.942076 32170 raft_consensus.cc:482] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:17.942167 32170 raft_consensus.cc:3037] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:17.944198 32170 raft_consensus.cc:504] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.944702 32170 leader_election.cc:288] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 election: Requested vote from peers 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:17.945099 32040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ef2fe052114dc993010e9ae13d3ca2" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27758f421417413b94fc31bacf4c3255"
I20260809 15:01:17.945140 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ef2fe052114dc993010e9ae13d3ca2" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
I20260809 15:01:17.945298 32040 raft_consensus.cc:3037] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:17.945309 31801 raft_consensus.cc:3037] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:17.948342 32040 raft_consensus.cc:2454] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 1.
I20260809 15:01:17.948342 31801 raft_consensus.cc:2454] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 1.
I20260809 15:01:17.948711 31422 leader_election.cc:302] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4aaf55d84e9d9640e1afc016da0b, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:17.948953 32170 raft_consensus.cc:2781] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:17.949353 32170 raft_consensus.cc:686] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 LEADER]: Becoming Leader. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Running, Role: LEADER
I20260809 15:01:17.949784 32170 consensus_queue.cc:237] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } }
I20260809 15:01:17.953917 31287 catalog_manager.cc:5495] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129). New cstate: current_term: 1 leader_uuid: "103c4aaf55d84e9d9640e1afc016da0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } health_report { overall_health: UNKNOWN } } }
W20260809 15:01:17.994357 32134 tablet.cc:2369] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 15:01:18 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:18.015929 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.004104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46089 (local address 127.0.0.1:56734)
0809 15:01:18.004656 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.004671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.004793 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.005077 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.005081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.005681 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:18.005950 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.005959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.006919 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.006923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.007873 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.007884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.008538 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.008543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.008652 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.009242 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:18.009259 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:18.011276 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:18.013775 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:18.013785 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:18.013790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:18.014217 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:18.014534 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:18.014537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:18.014540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:18.014684 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:18.015158 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:18.015163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:18.015296 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.015421 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.015600 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
I20260809 15:01:18.015986 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.004030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.27.186.190:46089)
0809 15:01:18.004205 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.004216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.004749 (+ 533us) server_negotiation.cc:407] Connection header received
0809 15:01:18.004875 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.004878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.004965 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.005039 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:18.006092 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.006801 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.007996 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.008394 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.008482 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.011436 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:18.011441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:18.011444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:18.011472 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:18.013659 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:18.014310 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:18.014314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:18.014316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:18.014447 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:18.014787 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:18.014791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:18.014793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:18.015053 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:18.015155 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.015520 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.015721 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":58}
I20260809 15:01:18.019914 31287 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:56734:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 15:01:18.020870 31287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:18.027088 31501 tablet_service.cc:1467] Processing CreateTablet for tablet ef887b58949241cfaab310f537d6d90c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cf242f1afef4c98ad65044ce944600f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:01:18.027235 31756 tablet_service.cc:1467] Processing CreateTablet for tablet ef887b58949241cfaab310f537d6d90c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cf242f1afef4c98ad65044ce944600f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:01:18.027942 31501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef887b58949241cfaab310f537d6d90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:18.027576 32003 tablet_service.cc:1467] Processing CreateTablet for tablet ef887b58949241cfaab310f537d6d90c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cf242f1afef4c98ad65044ce944600f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:01:18.028098 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef887b58949241cfaab310f537d6d90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:18.028368 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef887b58949241cfaab310f537d6d90c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 15:01:18.033476 31886 tablet.cc:2369] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:18.033941 32161 tablet_bootstrap.cc:492] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: Bootstrap starting.
I20260809 15:01:18.034976 32160 tablet_bootstrap.cc:492] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: Bootstrap starting.
I20260809 15:01:18.035390 32162 tablet_bootstrap.cc:492] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: Bootstrap starting.
I20260809 15:01:18.035439 32161 tablet_bootstrap.cc:654] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:18.036554 32160 tablet_bootstrap.cc:654] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:18.036814 32162 tablet_bootstrap.cc:654] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:18.037642 32161 tablet_bootstrap.cc:492] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: No bootstrap required, opened a new log
I20260809 15:01:18.037796 32161 ts_tablet_manager.cc:1397] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 15:01:18.038606 32161 raft_consensus.cc:348] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.038853 32161 raft_consensus.cc:374] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:18.038908 32160 tablet_bootstrap.cc:492] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: No bootstrap required, opened a new log
I20260809 15:01:18.039031 32160 ts_tablet_manager.cc:1397] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260809 15:01:18.039001 32161 raft_consensus.cc:729] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Initialized, Role: FOLLOWER
I20260809 15:01:18.039250 32161 consensus_queue.cc:260] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.039717 32170 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.039752 32160 raft_consensus.cc:348] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.039906 32161 ts_tablet_manager.cc:1428] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:18.039978 32160 raft_consensus.cc:374] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:18.040027 32170 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.040060 32160 raft_consensus.cc:729] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Initialized, Role: FOLLOWER
I20260809 15:01:18.040347 32160 consensus_queue.cc:260] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.040823 32169 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.041182 32169 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.041559 32162 tablet_bootstrap.cc:492] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: No bootstrap required, opened a new log
I20260809 15:01:18.041695 32162 ts_tablet_manager.cc:1397] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 15:01:18.042306 32160 ts_tablet_manager.cc:1428] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 15:01:18.042469 32162 raft_consensus.cc:348] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.042678 32162 raft_consensus.cc:374] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:18.042785 32162 raft_consensus.cc:729] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42a4d8c2de64aaeaf1ef7c0381e8efe, State: Initialized, Role: FOLLOWER
I20260809 15:01:18.043006 32162 consensus_queue.cc:260] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.043422 32171 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.043787 32171 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.044083 32162 ts_tablet_manager.cc:1428] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260809 15:01:18.051779 31627 tablet.cc:2369] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:18.168947 32169 raft_consensus.cc:482] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:18.169164 32169 raft_consensus.cc:504] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.170795 32169 leader_election.cc:288] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:18.176954 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.171041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:44207 (local address 127.27.186.131:40163)
0809 15:01:18.171346 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.171365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.171495 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.171855 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.171860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.171900 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:18.172133 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.172144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.173444 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.173448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.175296 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.175313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.176445 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.176449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.176582 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.176606 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.176676 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.176763 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260809 15:01:18.177100 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.171108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:40163 (local address 127.27.186.129:44207)
0809 15:01:18.171317 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.171323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.171441 (+ 118us) server_negotiation.cc:407] Connection header received
0809 15:01:18.171624 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.171630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.171711 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.171822 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:18.172267 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.173319 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.175444 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.176314 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.176464 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.176534 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.176716 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.176855 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:01:18.177997 32198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.171219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33527 (local address 127.27.186.131:55367)
0809 15:01:18.171980 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.171997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.172119 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.172438 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.172441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.172461 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:18.172760 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.172769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.174151 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.174157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.175899 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.175911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.177379 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.177386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.177549 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.177579 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.177668 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.177762 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":479,"threads_started":1}
I20260809 15:01:18.178104 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.171286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:55367 (local address 127.27.186.130:33527)
0809 15:01:18.171443 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.171448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.172081 (+ 633us) server_negotiation.cc:407] Connection header received
0809 15:01:18.172232 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.172237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.172328 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.172422 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:18.172924 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.173993 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.176049 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.177155 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.177317 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.177387 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.177696 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.177813 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:01:18.178058 31538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef887b58949241cfaab310f537d6d90c" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4aaf55d84e9d9640e1afc016da0b" is_pre_election: true
I20260809 15:01:18.178495 31538 raft_consensus.cc:2454] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 0.
I20260809 15:01:18.178695 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef887b58949241cfaab310f537d6d90c" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" is_pre_election: true
I20260809 15:01:18.178938 31801 raft_consensus.cc:2454] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 0.
I20260809 15:01:18.179158 31915 leader_election.cc:302] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4aaf55d84e9d9640e1afc016da0b, 27758f421417413b94fc31bacf4c3255; no voters:
I20260809 15:01:18.179571 32169 raft_consensus.cc:2781] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:18.179697 32169 raft_consensus.cc:482] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:18.179800 32169 raft_consensus.cc:3037] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:18.181989 32169 raft_consensus.cc:504] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.182542 32169 leader_election.cc:288] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 election: Requested vote from peers 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:18.182957 31538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef887b58949241cfaab310f537d6d90c" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4aaf55d84e9d9640e1afc016da0b"
I20260809 15:01:18.183004 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef887b58949241cfaab310f537d6d90c" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
I20260809 15:01:18.183145 31538 raft_consensus.cc:3037] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:18.183192 31801 raft_consensus.cc:3037] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:18.185073 31801 raft_consensus.cc:2454] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 1.
I20260809 15:01:18.185174 31538 raft_consensus.cc:2454] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 1.
I20260809 15:01:18.185420 31911 leader_election.cc:302] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27758f421417413b94fc31bacf4c3255, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:18.185670 32169 raft_consensus.cc:2781] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:18.186084 32169 raft_consensus.cc:686] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [term 1 LEADER]: Becoming Leader. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Running, Role: LEADER
I20260809 15:01:18.186544 32169 consensus_queue.cc:237] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.189867 32199 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: TxnStatusTablet state changed. Reason: New leader 27758f421417413b94fc31bacf4c3255. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.190285 32199 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:01:18.190932 32202 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 15:01:18.191193 32202 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 15:01:18.191349 31288 catalog_manager.cc:5495] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27758f421417413b94fc31bacf4c3255 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } health_report { overall_health: HEALTHY } } }
I20260809 15:01:18.230773 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.224893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39711 (local address 127.0.0.1:48574)
0809 15:01:18.225121 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.225136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.225288 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.225725 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.225730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.225742 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:18.226081 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.226100 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.227210 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.227216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.228226 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.228239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.229001 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.229006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.229178 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.229241 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.230215 (+ 974us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.230232 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.230303 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.230460 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 15:01:18.230808 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.224820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.27.186.131:39711)
0809 15:01:18.224975 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.224981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.225207 (+ 226us) server_negotiation.cc:407] Connection header received
0809 15:01:18.225476 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.225480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.225556 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.225674 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:18.226245 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.227072 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.228370 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.228842 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.228957 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.229418 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.230011 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.230167 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.230344 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.230430 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 15:01:18.239017 31801 raft_consensus.cc:1261] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Refusing update from remote peer 27758f421417413b94fc31bacf4c3255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:18.239159 31538 raft_consensus.cc:1261] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Refusing update from remote peer 27758f421417413b94fc31bacf4c3255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:18.239933 32169 consensus_queue.cc:1035] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:18.240296 32199 consensus_queue.cc:1035] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:18.250944 32171 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: TxnStatusTablet state changed. Reason: New leader 27758f421417413b94fc31bacf4c3255. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.251221 32171 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.253116 32170 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: TxnStatusTablet state changed. Reason: New leader 27758f421417413b94fc31bacf4c3255. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.253436 32170 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.253610 32208 mvcc.cc:205] Tried to move back new op lower bound from 7316634330058461184 to 7316634329855758336. Current Snapshot: MvccSnapshot[applied={T|T < 7316634330058461184}]
I20260809 15:01:18.254930 32211 mvcc.cc:205] Tried to move back new op lower bound from 7316634330058461184 to 7316634329855758336. Current Snapshot: MvccSnapshot[applied={T|T < 7316634330058461184}]
I20260809 15:01:18.256033 32209 mvcc.cc:205] Tried to move back new op lower bound from 7316634330058461184 to 7316634329855758336. Current Snapshot: MvccSnapshot[applied={T|T < 7316634330058461184}]
I20260809 15:01:18.262562 32170 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.262823 32170 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 103c4aaf55d84e9d9640e1afc016da0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.264361 32169 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.264593 32169 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:01:18.264950 32169 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.265161 32169 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P 27758f421417413b94fc31bacf4c3255: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:01:18.266623 32171 tablet_replica.cc:439] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } } }
I20260809 15:01:18.266849 32171 tablet_replica.cc:442] T ef887b58949241cfaab310f537d6d90c P d42a4d8c2de64aaeaf1ef7c0381e8efe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:01:18.278316 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.272482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.27.186.129:44207)
0809 15:01:18.272650 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.272656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.272850 (+ 194us) server_negotiation.cc:407] Connection header received
0809 15:01:18.273037 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.273040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.273108 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.273233 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:18.273601 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.274472 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.276332 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.276703 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.276788 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.277314 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.277749 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.277853 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.277962 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.278049 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:01:18.296602 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.291321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:53911 (local address 127.27.186.131:39711)
0809 15:01:18.291687 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.291693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.292071 (+ 378us) server_negotiation.cc:407] Connection header received
0809 15:01:18.292246 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.292251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.292312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.292384 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:18.292846 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.293570 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.294916 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.295358 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.295470 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.295879 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.295976 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.296047 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.296234 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.296310 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260809 15:01:18.297305 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.291355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39711 (local address 127.27.186.129:53911)
0809 15:01:18.291975 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.291990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.292119 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.292408 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.292413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.292425 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:18.292706 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.292714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.293695 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.293699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.294775 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.294787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.295525 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.295532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.295645 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.295700 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.296066 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.296078 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.296823 (+ 745us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.296949 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":347,"threads_started":1}
I20260809 15:01:18.312516 31801 raft_consensus.cc:1261] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Refusing update from remote peer 103c4aaf55d84e9d9640e1afc016da0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:18.312892 32040 raft_consensus.cc:1261] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Refusing update from remote peer 103c4aaf55d84e9d9640e1afc016da0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:18.313100 32174 consensus_queue.cc:1035] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:18.313503 32170 consensus_queue.cc:1035] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:18.344542 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.338067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37109 (local address 127.27.186.129:44207)
0809 15:01:18.338251 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.338258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.339324 (+ 1066us) server_negotiation.cc:407] Connection header received
0809 15:01:18.339496 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.339502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.339565 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.339634 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:18.340171 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.341062 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.342912 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.343331 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.343439 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.343850 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.343961 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.344029 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.344176 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.344263 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:01:18.344645 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.338162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:44207 (local address 127.27.186.131:37109)
0809 15:01:18.339229 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.339244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.339376 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.339663 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.339668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.339683 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:18.340007 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.340017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.341234 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.341239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.342778 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.342791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.343512 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.343519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.343640 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.343700 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.344054 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.344074 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.344146 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.344270 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":787,"threads_started":1}
I20260809 15:01:18.382854 31288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:18.383127 31288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:18.386821 31288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Sending DeleteTablet for 3 replicas of tablet e7ef2fe052114dc993010e9ae13d3ca2
I20260809 15:01:18.387799 32003 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ef2fe052114dc993010e9ae13d3ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:18 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:47510
I20260809 15:01:18.388456 31501 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ef2fe052114dc993010e9ae13d3ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:18 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:37592
I20260809 15:01:18.388743 32238 tablet_replica.cc:330] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: stopping tablet replica
I20260809 15:01:18.389179 32238 raft_consensus.cc:2229] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:18.389567 32238 raft_consensus.cc:2258] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:18.390403 31756 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ef2fe052114dc993010e9ae13d3ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:18 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:59114
I20260809 15:01:18.391433 31288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 15:01:18.391709 32238 ts_tablet_manager.cc:1905] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 15:01:18.392031 31288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:18.399302 32238 ts_tablet_manager.cc:1918] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:18.399471 32238 log.cc:1196] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/e7ef2fe052114dc993010e9ae13d3ca2
I20260809 15:01:18.400002 32238 ts_tablet_manager.cc:1939] T e7ef2fe052114dc993010e9ae13d3ca2 P 27758f421417413b94fc31bacf4c3255: Deleting consensus metadata
I20260809 15:01:18.401633 31258 catalog_manager.cc:4841] TS 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711): tablet e7ef2fe052114dc993010e9ae13d3ca2 (table test-table [id=8ee6f3419d074e02b2a7c80b49e3ba61]) successfully deleted
I20260809 15:01:18.404028 31500 tablet_service.cc:1467] Processing CreateTablet for tablet 78490676f0244a3d83dcc5d51bce7e12 (DEFAULT_TABLE table=test-table [id=609d123f8837489a8592f60427d07e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:18.404554 31500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78490676f0244a3d83dcc5d51bce7e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:18.404610 31754 tablet_service.cc:1467] Processing CreateTablet for tablet 78490676f0244a3d83dcc5d51bce7e12 (DEFAULT_TABLE table=test-table [id=609d123f8837489a8592f60427d07e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:18.405083 31754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78490676f0244a3d83dcc5d51bce7e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:18.410766 32003 tablet_service.cc:1467] Processing CreateTablet for tablet 78490676f0244a3d83dcc5d51bce7e12 (DEFAULT_TABLE table=test-table [id=609d123f8837489a8592f60427d07e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:18.411181 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78490676f0244a3d83dcc5d51bce7e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:18.411427 32161 tablet_bootstrap.cc:492] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Bootstrap starting.
I20260809 15:01:18.413125 32161 tablet_bootstrap.cc:654] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:18.421017 32240 tablet_replica.cc:330] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: stopping tablet replica
I20260809 15:01:18.421060 32162 tablet_bootstrap.cc:492] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Bootstrap starting.
I20260809 15:01:18.421416 32240 raft_consensus.cc:2229] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:18.422056 32240 raft_consensus.cc:2258] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:18.422688 32162 tablet_bootstrap.cc:654] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:18.422725 32242 tablet_replica.cc:330] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: stopping tablet replica
I20260809 15:01:18.423123 32242 raft_consensus.cc:2229] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:18.423466 32242 raft_consensus.cc:2258] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:18.425361 32242 ts_tablet_manager.cc:1905] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:18.426179 32160 tablet_bootstrap.cc:492] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Bootstrap starting.
I20260809 15:01:18.426501 32162 tablet_bootstrap.cc:492] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: No bootstrap required, opened a new log
I20260809 15:01:18.426659 32162 ts_tablet_manager.cc:1397] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260809 15:01:18.427461 32162 raft_consensus.cc:348] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.427675 32160 tablet_bootstrap.cc:654] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:18.427722 32162 raft_consensus.cc:374] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:18.427809 32162 raft_consensus.cc:729] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42a4d8c2de64aaeaf1ef7c0381e8efe, State: Initialized, Role: FOLLOWER
I20260809 15:01:18.428050 32162 consensus_queue.cc:260] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.428614 32162 ts_tablet_manager.cc:1428] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 15:01:18.431738 32240 ts_tablet_manager.cc:1905] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:18.437669 32242 ts_tablet_manager.cc:1918] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:18.437844 32242 log.cc:1196] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/e7ef2fe052114dc993010e9ae13d3ca2
I20260809 15:01:18.438350 32242 ts_tablet_manager.cc:1939] T e7ef2fe052114dc993010e9ae13d3ca2 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting consensus metadata
I20260809 15:01:18.439744 32240 ts_tablet_manager.cc:1918] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:18.439926 32240 log.cc:1196] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/e7ef2fe052114dc993010e9ae13d3ca2
I20260809 15:01:18.439958 31258 catalog_manager.cc:4841] TS d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527): tablet e7ef2fe052114dc993010e9ae13d3ca2 (table test-table [id=8ee6f3419d074e02b2a7c80b49e3ba61]) successfully deleted
I20260809 15:01:18.440610 32240 ts_tablet_manager.cc:1939] T e7ef2fe052114dc993010e9ae13d3ca2 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting consensus metadata
I20260809 15:01:18.441692 32160 tablet_bootstrap.cc:492] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: No bootstrap required, opened a new log
I20260809 15:01:18.441867 32160 ts_tablet_manager.cc:1397] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260809 15:01:18.442237 31259 catalog_manager.cc:4841] TS 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207): tablet e7ef2fe052114dc993010e9ae13d3ca2 (table test-table [id=8ee6f3419d074e02b2a7c80b49e3ba61]) successfully deleted
I20260809 15:01:18.442642 32160 raft_consensus.cc:348] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.442890 32160 raft_consensus.cc:374] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:18.442982 32160 raft_consensus.cc:729] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Initialized, Role: FOLLOWER
I20260809 15:01:18.443310 32160 consensus_queue.cc:260] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.444448 32161 tablet_bootstrap.cc:492] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: No bootstrap required, opened a new log
I20260809 15:01:18.444638 32161 ts_tablet_manager.cc:1397] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.002s
I20260809 15:01:18.445421 32161 raft_consensus.cc:348] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.445665 32161 raft_consensus.cc:374] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:18.445785 32161 raft_consensus.cc:729] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Initialized, Role: FOLLOWER
I20260809 15:01:18.446045 32160 ts_tablet_manager.cc:1428] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 15:01:18.446049 32161 consensus_queue.cc:260] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.446605 32161 ts_tablet_manager.cc:1428] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:18.606160 32227 raft_consensus.cc:482] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:18.606393 32227 raft_consensus.cc:504] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.607245 32227 leader_election.cc:288] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:18.607843 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78490676f0244a3d83dcc5d51bce7e12" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4aaf55d84e9d9640e1afc016da0b" is_pre_election: true
I20260809 15:01:18.607992 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78490676f0244a3d83dcc5d51bce7e12" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" is_pre_election: true
I20260809 15:01:18.608132 31536 raft_consensus.cc:2454] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 0.
I20260809 15:01:18.608243 31801 raft_consensus.cc:2454] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 0.
I20260809 15:01:18.608724 31911 leader_election.cc:302] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27758f421417413b94fc31bacf4c3255, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:18.609021 32227 raft_consensus.cc:2781] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:18.609146 32227 raft_consensus.cc:482] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:18.609263 32227 raft_consensus.cc:3037] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:18.612576 32227 raft_consensus.cc:504] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.613757 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78490676f0244a3d83dcc5d51bce7e12" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4aaf55d84e9d9640e1afc016da0b"
I20260809 15:01:18.613807 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78490676f0244a3d83dcc5d51bce7e12" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
I20260809 15:01:18.613976 31536 raft_consensus.cc:3037] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:18.613982 31801 raft_consensus.cc:3037] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:18.616986 31801 raft_consensus.cc:2454] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 1.
I20260809 15:01:18.617292 31536 raft_consensus.cc:2454] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 1.
I20260809 15:01:18.617415 31911 leader_election.cc:302] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27758f421417413b94fc31bacf4c3255, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:18.617762 32169 raft_consensus.cc:2781] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:18.617941 32169 raft_consensus.cc:686] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 1 LEADER]: Becoming Leader. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Running, Role: LEADER
I20260809 15:01:18.618247 32169 consensus_queue.cc:237] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:18.618615 32227 leader_election.cc:288] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 election: Requested vote from peers 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:18.621173 31283 catalog_manager.cc:5495] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27758f421417413b94fc31bacf4c3255 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } health_report { overall_health: HEALTHY } } }
I20260809 15:01:18.672432 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.667170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.27.186.131:39711)
0809 15:01:18.667358 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.667364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.667382 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:01:18.667441 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.667446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.667532 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.667626 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:18.668061 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.669241 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.670650 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.671292 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.671425 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.671739 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.671883 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.671974 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.672059 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.672144 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:01:18.685055 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.680162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39711 (local address 127.27.186.131:55247)
0809 15:01:18.680320 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:18.680336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:18.680446 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:18.680733 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:18.680738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:18.680750 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:01:18.680892 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.680898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.682223 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.682227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.683232 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:18.683244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.684143 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.684147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:18.684277 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.684310 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.684658 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.684664 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:01:18.684715 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 15:01:18.684806 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260809 15:01:18.685124 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:18.680243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:55247 (local address 127.27.186.131:39711)
0809 15:01:18.680408 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:18.680413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:18.680432 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:18.680549 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:18.680554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:18.680624 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:18.680722 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:18.681010 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.682095 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.683366 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:18.684021 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:18.684143 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:18.684463 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:18.684577 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:18.684665 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:18.684748 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 15:01:18.684817 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:01:18.693053 31536 raft_consensus.cc:1261] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Refusing update from remote peer 27758f421417413b94fc31bacf4c3255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:18.693101 31801 raft_consensus.cc:1261] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Refusing update from remote peer 27758f421417413b94fc31bacf4c3255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:18.693535 32227 consensus_queue.cc:1035] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:18.693768 32169 consensus_queue.cc:1035] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:01:18.720542 32208 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 15:01:20.313143 31283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:20.313380 31283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:20.317194 31283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Sending DeleteTablet for 3 replicas of tablet 78490676f0244a3d83dcc5d51bce7e12
I20260809 15:01:20.317853 31501 tablet_service.cc:1514] Processing DeleteTablet for tablet 78490676f0244a3d83dcc5d51bce7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:20 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:37592
I20260809 15:01:20.317929 31756 tablet_service.cc:1514] Processing DeleteTablet for tablet 78490676f0244a3d83dcc5d51bce7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:20 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:59114
I20260809 15:01:20.318073 32003 tablet_service.cc:1514] Processing DeleteTablet for tablet 78490676f0244a3d83dcc5d51bce7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:20 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:47510
I20260809 15:01:20.330269 32287 tablet_replica.cc:330] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: stopping tablet replica
I20260809 15:01:20.330475 32284 tablet_replica.cc:330] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: stopping tablet replica
I20260809 15:01:20.330462 32287 raft_consensus.cc:2229] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:20.330675 32284 raft_consensus.cc:2229] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:20.330925 32284 raft_consensus.cc:2258] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:20.331791 32287 raft_consensus.cc:2258] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:20.332017 32284 ts_tablet_manager.cc:1905] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:20.333164 32286 tablet_replica.cc:330] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: stopping tablet replica
I20260809 15:01:20.333290 32287 ts_tablet_manager.cc:1905] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:20.333405 32286 raft_consensus.cc:2229] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:20.333691 32286 raft_consensus.cc:2258] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:20.334590 32286 ts_tablet_manager.cc:1905] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:20.338510 32284 ts_tablet_manager.cc:1918] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 15:01:20.338711 32284 log.cc:1196] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/78490676f0244a3d83dcc5d51bce7e12
I20260809 15:01:20.339287 32284 ts_tablet_manager.cc:1939] T 78490676f0244a3d83dcc5d51bce7e12 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting consensus metadata
I20260809 15:01:20.339956 32287 ts_tablet_manager.cc:1918] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 15:01:20.340143 32287 log.cc:1196] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/78490676f0244a3d83dcc5d51bce7e12
I20260809 15:01:20.340423 31259 catalog_manager.cc:4841] TS 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207): tablet 78490676f0244a3d83dcc5d51bce7e12 (table test-table [id=609d123f8837489a8592f60427d07e3d]) successfully deleted
I20260809 15:01:20.340646 32287 ts_tablet_manager.cc:1939] T 78490676f0244a3d83dcc5d51bce7e12 P 27758f421417413b94fc31bacf4c3255: Deleting consensus metadata
I20260809 15:01:20.340788 32286 ts_tablet_manager.cc:1918] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 15:01:20.340947 32286 log.cc:1196] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/78490676f0244a3d83dcc5d51bce7e12
I20260809 15:01:20.341401 32286 ts_tablet_manager.cc:1939] T 78490676f0244a3d83dcc5d51bce7e12 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting consensus metadata
I20260809 15:01:20.341738 31258 catalog_manager.cc:4841] TS 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711): tablet 78490676f0244a3d83dcc5d51bce7e12 (table test-table [id=609d123f8837489a8592f60427d07e3d]) successfully deleted
I20260809 15:01:20.342476 31258 catalog_manager.cc:4841] TS d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527): tablet 78490676f0244a3d83dcc5d51bce7e12 (table test-table [id=609d123f8837489a8592f60427d07e3d]) successfully deleted
Aug 09 15:01:20 dist-test-slave-q9h9 krb5kdc[31206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 15:01:20 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:20.374574 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:20.361630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.27.186.190:46089)
0809 15:01:20.362207 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:20.362212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:20.362232 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:20.362356 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:20.362360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:20.362421 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:20.362498 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:20.363742 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:20.364523 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:20.365680 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:20.366160 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:20.366300 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:20.369808 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:20.369813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:20.369817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:20.369848 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:20.372336 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:20.372889 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:20.372895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:20.372898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:20.373013 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:20.373287 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:20.373291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:20.373293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:20.373556 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:20.373733 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:20.373983 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 15:01:20.374181 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":252,"threads_started":1}
I20260809 15:01:20.377504 31283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:20.378096 31283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:20.384141 31501 tablet_service.cc:1467] Processing CreateTablet for tablet c489cfc35c0d46f0acbc39ab510b2cad (DEFAULT_TABLE table=test-table [id=8fa207176f6045b791b5baa37988595b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:20.384416 32003 tablet_service.cc:1467] Processing CreateTablet for tablet c489cfc35c0d46f0acbc39ab510b2cad (DEFAULT_TABLE table=test-table [id=8fa207176f6045b791b5baa37988595b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:20.384565 31501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c489cfc35c0d46f0acbc39ab510b2cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:20.384833 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c489cfc35c0d46f0acbc39ab510b2cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:20.386401 31756 tablet_service.cc:1467] Processing CreateTablet for tablet c489cfc35c0d46f0acbc39ab510b2cad (DEFAULT_TABLE table=test-table [id=8fa207176f6045b791b5baa37988595b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:20.386807 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c489cfc35c0d46f0acbc39ab510b2cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:20.391080 32299 tablet_bootstrap.cc:492] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Bootstrap starting.
I20260809 15:01:20.391477 32301 tablet_bootstrap.cc:492] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Bootstrap starting.
I20260809 15:01:20.392695 32299 tablet_bootstrap.cc:654] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:20.392766 32303 tablet_bootstrap.cc:492] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Bootstrap starting.
I20260809 15:01:20.392896 32301 tablet_bootstrap.cc:654] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:20.394639 32303 tablet_bootstrap.cc:654] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:20.394652 32299 tablet_bootstrap.cc:492] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: No bootstrap required, opened a new log
I20260809 15:01:20.394783 32299 ts_tablet_manager.cc:1397] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 15:01:20.395593 32299 raft_consensus.cc:348] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.395840 32299 raft_consensus.cc:374] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:20.395917 32299 raft_consensus.cc:729] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Initialized, Role: FOLLOWER
I20260809 15:01:20.396111 32299 consensus_queue.cc:260] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.396894 32299 ts_tablet_manager.cc:1428] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:20.398311 32303 tablet_bootstrap.cc:492] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: No bootstrap required, opened a new log
I20260809 15:01:20.398339 32301 tablet_bootstrap.cc:492] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: No bootstrap required, opened a new log
I20260809 15:01:20.398469 32301 ts_tablet_manager.cc:1397] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260809 15:01:20.398514 32303 ts_tablet_manager.cc:1397] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:01:20.399216 32303 raft_consensus.cc:348] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.399216 32301 raft_consensus.cc:348] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.399452 32301 raft_consensus.cc:374] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:20.399519 32303 raft_consensus.cc:374] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:20.399549 32301 raft_consensus.cc:729] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Initialized, Role: FOLLOWER
I20260809 15:01:20.399611 32303 raft_consensus.cc:729] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42a4d8c2de64aaeaf1ef7c0381e8efe, State: Initialized, Role: FOLLOWER
I20260809 15:01:20.399760 32301 consensus_queue.cc:260] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.399880 32303 consensus_queue.cc:260] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.400303 32301 ts_tablet_manager.cc:1428] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:20.400804 32303 ts_tablet_manager.cc:1428] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:20.556071 32274 raft_consensus.cc:482] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:20.556229 32274 raft_consensus.cc:504] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.556797 32274 leader_election.cc:288] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:20.557278 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c489cfc35c0d46f0acbc39ab510b2cad" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4aaf55d84e9d9640e1afc016da0b" is_pre_election: true
I20260809 15:01:20.557325 31799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c489cfc35c0d46f0acbc39ab510b2cad" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" is_pre_election: true
I20260809 15:01:20.557488 31536 raft_consensus.cc:2454] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 0.
I20260809 15:01:20.557559 31799 raft_consensus.cc:2454] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 0.
I20260809 15:01:20.557930 31915 leader_election.cc:302] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4aaf55d84e9d9640e1afc016da0b, 27758f421417413b94fc31bacf4c3255; no voters:
I20260809 15:01:20.558218 32274 raft_consensus.cc:2781] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:20.558331 32274 raft_consensus.cc:482] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:20.558418 32274 raft_consensus.cc:3037] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:20.560607 32274 raft_consensus.cc:504] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.561128 32274 leader_election.cc:288] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 election: Requested vote from peers 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207), d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527)
I20260809 15:01:20.561508 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c489cfc35c0d46f0acbc39ab510b2cad" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4aaf55d84e9d9640e1afc016da0b"
I20260809 15:01:20.561596 31799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c489cfc35c0d46f0acbc39ab510b2cad" candidate_uuid: "27758f421417413b94fc31bacf4c3255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
I20260809 15:01:20.561733 31536 raft_consensus.cc:3037] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:20.561772 31799 raft_consensus.cc:3037] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:20.564643 31799 raft_consensus.cc:2454] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 1.
I20260809 15:01:20.564682 31536 raft_consensus.cc:2454] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27758f421417413b94fc31bacf4c3255 in term 1.
I20260809 15:01:20.565025 31911 leader_election.cc:302] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27758f421417413b94fc31bacf4c3255, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:20.565279 32274 raft_consensus.cc:2781] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:20.565426 32274 raft_consensus.cc:686] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 1 LEADER]: Becoming Leader. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Running, Role: LEADER
I20260809 15:01:20.565735 32274 consensus_queue.cc:237] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.568046 31288 catalog_manager.cc:5495] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27758f421417413b94fc31bacf4c3255 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "27758f421417413b94fc31bacf4c3255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } health_report { overall_health: HEALTHY } } }
I20260809 15:01:20.590986 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:20.586306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.27.186.131:39711)
0809 15:01:20.586815 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:20.586821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:20.586841 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:20.586890 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:20.586893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:20.586948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:20.587028 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:20.587333 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:20.588131 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:20.589460 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:20.589939 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:20.590086 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:20.590433 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:20.590542 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:20.590607 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:20.590692 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 15:01:20.590748 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":305,"threads_started":1}
I20260809 15:01:20.598332 31799 raft_consensus.cc:1261] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Refusing update from remote peer 27758f421417413b94fc31bacf4c3255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:20.598385 31536 raft_consensus.cc:1261] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Refusing update from remote peer 27758f421417413b94fc31bacf4c3255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:20.598805 32274 consensus_queue.cc:1035] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:20.599040 32257 consensus_queue.cc:1035] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:20.640157 31287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:20.640365 31287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:20.642874 31287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Sending DeleteTablet for 3 replicas of tablet c489cfc35c0d46f0acbc39ab510b2cad
I20260809 15:01:20.643537 31501 tablet_service.cc:1514] Processing DeleteTablet for tablet c489cfc35c0d46f0acbc39ab510b2cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:20 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:37592
I20260809 15:01:20.643561 31756 tablet_service.cc:1514] Processing DeleteTablet for tablet c489cfc35c0d46f0acbc39ab510b2cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:20 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:59114
I20260809 15:01:20.643662 32003 tablet_service.cc:1514] Processing DeleteTablet for tablet c489cfc35c0d46f0acbc39ab510b2cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:20 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:47510
I20260809 15:01:20.643800 32286 tablet_replica.cc:330] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: stopping tablet replica
I20260809 15:01:20.643977 32287 tablet_replica.cc:330] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: stopping tablet replica
I20260809 15:01:20.643986 32284 tablet_replica.cc:330] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: stopping tablet replica
I20260809 15:01:20.643991 32286 raft_consensus.cc:2229] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:20.644156 32286 raft_consensus.cc:2258] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:20.644153 32287 raft_consensus.cc:2229] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:20.644183 32284 raft_consensus.cc:2229] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:20.644357 32284 raft_consensus.cc:2258] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:20.644446 32287 raft_consensus.cc:2258] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:20.644510 31287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:20.645171 31287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:20.645378 32287 ts_tablet_manager.cc:1905] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:20.645378 32286 ts_tablet_manager.cc:1905] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:20.645496 32284 ts_tablet_manager.cc:1905] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:20.651723 32286 ts_tablet_manager.cc:1918] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:20.651741 32287 ts_tablet_manager.cc:1918] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:20.651872 32287 log.cc:1196] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/c489cfc35c0d46f0acbc39ab510b2cad
I20260809 15:01:20.652065 32286 log.cc:1196] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/c489cfc35c0d46f0acbc39ab510b2cad
I20260809 15:01:20.652348 32287 ts_tablet_manager.cc:1939] T c489cfc35c0d46f0acbc39ab510b2cad P 27758f421417413b94fc31bacf4c3255: Deleting consensus metadata
I20260809 15:01:20.652655 31501 tablet_service.cc:1467] Processing CreateTablet for tablet 5853c25281a04d0d9a2c62d72167b469 (DEFAULT_TABLE table=test-table [id=8b794733d8904f9199ee91793c814c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:20.652820 32003 tablet_service.cc:1467] Processing CreateTablet for tablet 5853c25281a04d0d9a2c62d72167b469 (DEFAULT_TABLE table=test-table [id=8b794733d8904f9199ee91793c814c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:20.653062 31501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5853c25281a04d0d9a2c62d72167b469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:20.653182 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5853c25281a04d0d9a2c62d72167b469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:20.653842 31258 catalog_manager.cc:4841] TS 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711): tablet c489cfc35c0d46f0acbc39ab510b2cad (table test-table [id=8fa207176f6045b791b5baa37988595b]) successfully deleted
I20260809 15:01:20.654865 32284 ts_tablet_manager.cc:1918] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:01:20.655031 32284 log.cc:1196] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/c489cfc35c0d46f0acbc39ab510b2cad
I20260809 15:01:20.655480 32284 ts_tablet_manager.cc:1939] T c489cfc35c0d46f0acbc39ab510b2cad P 103c4aaf55d84e9d9640e1afc016da0b: Deleting consensus metadata
I20260809 15:01:20.652859 31756 tablet_service.cc:1467] Processing CreateTablet for tablet 5853c25281a04d0d9a2c62d72167b469 (DEFAULT_TABLE table=test-table [id=8b794733d8904f9199ee91793c814c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:20.656478 31259 catalog_manager.cc:4841] TS 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207): tablet c489cfc35c0d46f0acbc39ab510b2cad (table test-table [id=8fa207176f6045b791b5baa37988595b]) successfully deleted
I20260809 15:01:20.656615 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5853c25281a04d0d9a2c62d72167b469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:20.656654 32286 ts_tablet_manager.cc:1939] T c489cfc35c0d46f0acbc39ab510b2cad P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting consensus metadata
I20260809 15:01:20.657606 31258 catalog_manager.cc:4841] TS d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527): tablet c489cfc35c0d46f0acbc39ab510b2cad (table test-table [id=8fa207176f6045b791b5baa37988595b]) successfully deleted
I20260809 15:01:20.658907 32301 tablet_bootstrap.cc:492] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Bootstrap starting.
I20260809 15:01:20.659027 32299 tablet_bootstrap.cc:492] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Bootstrap starting.
I20260809 15:01:20.660266 32301 tablet_bootstrap.cc:654] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:20.660562 32299 tablet_bootstrap.cc:654] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:20.662938 32303 tablet_bootstrap.cc:492] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Bootstrap starting.
I20260809 15:01:20.664104 32301 tablet_bootstrap.cc:492] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: No bootstrap required, opened a new log
I20260809 15:01:20.664263 32299 tablet_bootstrap.cc:492] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: No bootstrap required, opened a new log
I20260809 15:01:20.664361 32301 ts_tablet_manager.cc:1397] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 15:01:20.664445 32299 ts_tablet_manager.cc:1397] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260809 15:01:20.664580 32303 tablet_bootstrap.cc:654] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:20.665024 32299 raft_consensus.cc:348] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.665215 32299 raft_consensus.cc:374] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:20.665077 32301 raft_consensus.cc:348] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.665295 32299 raft_consensus.cc:729] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Initialized, Role: FOLLOWER
I20260809 15:01:20.665318 32301 raft_consensus.cc:374] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:20.665449 32301 raft_consensus.cc:729] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27758f421417413b94fc31bacf4c3255, State: Initialized, Role: FOLLOWER
I20260809 15:01:20.665537 32299 consensus_queue.cc:260] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.665797 32301 consensus_queue.cc:260] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.666127 32299 ts_tablet_manager.cc:1428] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:20.666981 32301 ts_tablet_manager.cc:1428] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 15:01:20.668675 32303 tablet_bootstrap.cc:492] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: No bootstrap required, opened a new log
I20260809 15:01:20.668834 32303 ts_tablet_manager.cc:1397] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260809 15:01:20.669579 32303 raft_consensus.cc:348] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.669871 32303 raft_consensus.cc:374] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:20.669983 32303 raft_consensus.cc:729] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42a4d8c2de64aaeaf1ef7c0381e8efe, State: Initialized, Role: FOLLOWER
I20260809 15:01:20.670284 32303 consensus_queue.cc:260] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.670747 32303 ts_tablet_manager.cc:1428] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:20.705752 32307 raft_consensus.cc:482] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:20.705932 32307 raft_consensus.cc:504] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.706543 32307 leader_election.cc:288] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527), 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711)
I20260809 15:01:20.706954 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853c25281a04d0d9a2c62d72167b469" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" is_pre_election: true
I20260809 15:01:20.707019 32040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853c25281a04d0d9a2c62d72167b469" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27758f421417413b94fc31bacf4c3255" is_pre_election: true
I20260809 15:01:20.707129 31801 raft_consensus.cc:2454] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 0.
I20260809 15:01:20.707230 32040 raft_consensus.cc:2454] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 0.
I20260809 15:01:20.707504 31422 leader_election.cc:302] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4aaf55d84e9d9640e1afc016da0b, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:20.707728 32307 raft_consensus.cc:2781] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:20.707829 32307 raft_consensus.cc:482] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:20.707917 32307 raft_consensus.cc:3037] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:20.709769 32307 raft_consensus.cc:504] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.710290 32307 leader_election.cc:288] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 election: Requested vote from peers d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527), 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711)
I20260809 15:01:20.710682 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853c25281a04d0d9a2c62d72167b469" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe"
I20260809 15:01:20.710721 32040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853c25281a04d0d9a2c62d72167b469" candidate_uuid: "103c4aaf55d84e9d9640e1afc016da0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27758f421417413b94fc31bacf4c3255"
I20260809 15:01:20.710839 31801 raft_consensus.cc:3037] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:20.710888 32040 raft_consensus.cc:3037] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:20.712677 31801 raft_consensus.cc:2454] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 1.
I20260809 15:01:20.712735 32040 raft_consensus.cc:2454] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103c4aaf55d84e9d9640e1afc016da0b in term 1.
I20260809 15:01:20.712996 31422 leader_election.cc:302] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4aaf55d84e9d9640e1afc016da0b, d42a4d8c2de64aaeaf1ef7c0381e8efe; no voters:
I20260809 15:01:20.713263 32307 raft_consensus.cc:2781] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:20.713617 32307 raft_consensus.cc:686] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 LEADER]: Becoming Leader. State: Replica: 103c4aaf55d84e9d9640e1afc016da0b, State: Running, Role: LEADER
I20260809 15:01:20.713876 32307 consensus_queue.cc:237] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } }
I20260809 15:01:20.716114 31288 catalog_manager.cc:5495] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129). New cstate: current_term: 1 leader_uuid: "103c4aaf55d84e9d9640e1afc016da0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103c4aaf55d84e9d9640e1afc016da0b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 44207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 } health_report { overall_health: UNKNOWN } } }
I20260809 15:01:20.733444 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:20.728664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.27.186.129:44207)
0809 15:01:20.729216 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:20.729220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:20.729243 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:01:20.729289 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:20.729293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:20.729348 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:20.729425 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:20.729759 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:20.730605 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:20.732015 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:20.732418 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:20.732510 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:20.732876 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:20.732966 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:20.733030 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:20.733123 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 15:01:20.733179 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":320,"threads_started":1}
I20260809 15:01:20.740746 32040 raft_consensus.cc:1261] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Refusing update from remote peer 103c4aaf55d84e9d9640e1afc016da0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:20.740789 31801 raft_consensus.cc:1261] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Refusing update from remote peer 103c4aaf55d84e9d9640e1afc016da0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:20.741278 32307 consensus_queue.cc:1035] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42a4d8c2de64aaeaf1ef7c0381e8efe" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:20.747771 32307 consensus_queue.cc:1035] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "27758f421417413b94fc31bacf4c3255" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:01:20.760808 32260 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 15:01:22.336571 31287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:22.336771 31287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:22.340512 31287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1051ec972e84c29967ed791d0643c11: Sending DeleteTablet for 3 replicas of tablet 5853c25281a04d0d9a2c62d72167b469
I20260809 15:01:22.341181 31501 tablet_service.cc:1514] Processing DeleteTablet for tablet 5853c25281a04d0d9a2c62d72167b469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:22 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:37592
I20260809 15:01:22.341286 31756 tablet_service.cc:1514] Processing DeleteTablet for tablet 5853c25281a04d0d9a2c62d72167b469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:22 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:59114
I20260809 15:01:22.341355 32003 tablet_service.cc:1514] Processing DeleteTablet for tablet 5853c25281a04d0d9a2c62d72167b469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:22 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:47510
I20260809 15:01:22.341944 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31398
I20260809 15:01:22.352304 32373 tablet_replica.cc:330] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: stopping tablet replica
I20260809 15:01:22.352569 32373 raft_consensus.cc:2229] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:22.352984 32373 raft_consensus.cc:2258] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:22.354385 32371 tablet_replica.cc:330] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: stopping tablet replica
I20260809 15:01:22.354513 32373 ts_tablet_manager.cc:1905] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:22.354605 32372 tablet_replica.cc:330] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: stopping tablet replica
I20260809 15:01:22.354786 32372 raft_consensus.cc:2229] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:22.355027 32372 raft_consensus.cc:2258] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:22.355355 32371 raft_consensus.cc:2229] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:22.356052 32372 ts_tablet_manager.cc:1905] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:22.356489 32371 raft_consensus.cc:2258] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:22.357352 32371 ts_tablet_manager.cc:1905] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:22.361179 32373 ts_tablet_manager.cc:1918] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 15:01:22.361347 32373 log.cc:1196] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/5853c25281a04d0d9a2c62d72167b469
I20260809 15:01:22.361763 32373 ts_tablet_manager.cc:1939] T 5853c25281a04d0d9a2c62d72167b469 P 27758f421417413b94fc31bacf4c3255: Deleting consensus metadata
I20260809 15:01:22.362370 32372 ts_tablet_manager.cc:1918] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 15:01:22.362504 32372 log.cc:1196] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/5853c25281a04d0d9a2c62d72167b469
I20260809 15:01:22.362675 32371 ts_tablet_manager.cc:1918] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 15:01:22.362841 32371 log.cc:1196] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/5853c25281a04d0d9a2c62d72167b469
I20260809 15:01:22.362808 31258 catalog_manager.cc:4841] TS 27758f421417413b94fc31bacf4c3255 (127.27.186.131:39711): tablet 5853c25281a04d0d9a2c62d72167b469 (table test-table [id=8b794733d8904f9199ee91793c814c92]) successfully deleted
I20260809 15:01:22.363158 32372 ts_tablet_manager.cc:1939] T 5853c25281a04d0d9a2c62d72167b469 P d42a4d8c2de64aaeaf1ef7c0381e8efe: Deleting consensus metadata
I20260809 15:01:22.363314 32371 ts_tablet_manager.cc:1939] T 5853c25281a04d0d9a2c62d72167b469 P 103c4aaf55d84e9d9640e1afc016da0b: Deleting consensus metadata
I20260809 15:01:22.364194 31259 catalog_manager.cc:4841] TS 103c4aaf55d84e9d9640e1afc016da0b (127.27.186.129:44207): tablet 5853c25281a04d0d9a2c62d72167b469 (table test-table [id=8b794733d8904f9199ee91793c814c92]) successfully deleted
I20260809 15:01:22.364219 31258 catalog_manager.cc:4841] TS d42a4d8c2de64aaeaf1ef7c0381e8efe (127.27.186.130:33527): tablet 5853c25281a04d0d9a2c62d72167b469 (table test-table [id=8b794733d8904f9199ee91793c814c92]) successfully deleted
Aug 09 15:01:22 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:22.372624 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:22.343252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.27.186.129:44207)
0809 15:01:22.351216 (+ 7964us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:22.351223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:22.352546 (+ 1323us) server_negotiation.cc:407] Connection header received
0809 15:01:22.352610 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:22.352616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:22.352761 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:22.352860 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:22.354760 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:22.355768 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:22.357600 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:22.358299 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:22.358414 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:22.367368 (+ 8954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:22.367381 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:22.367397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:22.367432 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:22.370063 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:22.370660 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:22.370667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:22.370669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:22.370805 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:22.371162 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:22.371165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:22.371167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:22.371522 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:22.371808 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:22.372097 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 15:01:22.372316 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7814,"thread_start_us":7681,"threads_started":1}
W20260809 15:01:22.373219 31615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48104
W20260809 15:01:22.373770 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31398: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:22.373961 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31398
I20260809 15:01:22.402632 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31630
Aug 09 15:01:22 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:22.416584 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:22.403279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.27.186.130:33527)
0809 15:01:22.403768 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:22.403772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:22.403786 (+ 14us) server_negotiation.cc:407] Connection header received
0809 15:01:22.403833 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:22.403836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:22.403890 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:22.403982 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:22.405152 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:22.405922 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:22.407127 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:22.407538 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:22.407645 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:22.411364 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:22.411377 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:22.411392 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:22.411428 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:22.413910 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:22.414575 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:22.414582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:22.414584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:22.414730 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:22.415089 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:22.415093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:22.415095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:22.415459 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:22.415731 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:22.416051 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 15:01:22.416279 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
W20260809 15:01:22.417227 31873 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40038
W20260809 15:01:22.417802 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31630: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:22.417932 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31630
I20260809 15:01:22.439533 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31889
Aug 09 15:01:22 dist-test-slave-q9h9 krb5kdc[31206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:22.453140 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:22.440355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.27.186.131:39711)
0809 15:01:22.440809 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:22.440813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:22.440828 (+ 15us) server_negotiation.cc:407] Connection header received
0809 15:01:22.440878 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:22.440882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:22.440937 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:22.441018 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:22.442143 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:22.442855 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:22.444100 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:22.444556 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:22.444673 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:22.448514 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:22.448528 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:22.448543 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:22.448580 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:22.451036 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:22.451607 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:22.451612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:22.451615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:22.451737 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:22.451996 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:22.451998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:22.452000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:22.452340 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:22.452562 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:22.452736 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 15:01:22.452897 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":213,"threads_started":1}
W20260809 15:01:22.453794 32121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606
W20260809 15:01:22.454368 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31889: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:22.454509 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31889
I20260809 15:01:22.482317 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31228
I20260809 15:01:22.483325 31347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:22.594420 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31228
2026-08-09T15:01:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8888 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:22 dist-test-slave-q9h9 krb5kdc[32385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:22 dist-test-slave-q9h9 krb5kdc[32385](info): set up 2 sockets
Aug 09 15:01:22 dist-test-slave-q9h9 krb5kdc[32385](info): commencing operation
krb5kdc: starting...
W20260809 15:01:23.789724 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:23 dist-test-slave-q9h9 krb5kdc[32385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:23.958992 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33467
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:24.261229 32400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:24.261508 32400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:24.261641 32400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:24.268543 32400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:24.268641 32400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:24.268719 32400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:24.268796 32400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:24.268862 32400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:24.278097 32400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:24.279176 32400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:24.280339 32400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:24.291178 32412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:24.291304 32410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:24.291205 32417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:24.291898 32400 server_base.cc:1034] running on GCE node
I20260809 15:01:24.292585 32400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:24.293854 32400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:24.295053 32400 hybrid_clock.cc:648] HybridClock initialized: now 1786287684294992 us; error 47 us; skew 500 ppm
I20260809 15:01:24.295365 32400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:24 dist-test-slave-q9h9 krb5kdc[32385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:24.298220 32400 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:24.299502 32400 webserver.cc:460] Webserver started at http://127.27.186.190:39129/ using document root <none> and password file <none>
I20260809 15:01:24.300125 32400 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:24.300239 32400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:24.300520 32400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:24.303262 32400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "6cfbe2b13bf94b368dc43417cd36409a"
format_stamp: "Formatted at 2026-08-09 15:01:24 on dist-test-slave-q9h9"
server_key: "1883016777ba35d4542de03772bf9e3b"
server_key_iv: "30f5e97dbfbfd46a244cbf2190409b9e"
server_key_version: "encryptionkey@0"
I20260809 15:01:24.303966 32400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "6cfbe2b13bf94b368dc43417cd36409a"
format_stamp: "Formatted at 2026-08-09 15:01:24 on dist-test-slave-q9h9"
server_key: "1883016777ba35d4542de03772bf9e3b"
server_key_iv: "30f5e97dbfbfd46a244cbf2190409b9e"
server_key_version: "encryptionkey@0"
I20260809 15:01:24.308898 32400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 15:01:24.312105 32425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:24.313269 32400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:24.313421 32400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "6cfbe2b13bf94b368dc43417cd36409a"
format_stamp: "Formatted at 2026-08-09 15:01:24 on dist-test-slave-q9h9"
server_key: "1883016777ba35d4542de03772bf9e3b"
server_key_iv: "30f5e97dbfbfd46a244cbf2190409b9e"
server_key_version: "encryptionkey@0"
I20260809 15:01:24.313585 32400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:24.333899 32400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:24.337401 32400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:24.337632 32400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:24.353767 32400 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:33467
I20260809 15:01:24.353801 32514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:33467 every 8 connection(s)
I20260809 15:01:24.355031 32400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:24.358389 32515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:24.362517 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 32400
I20260809 15:01:24.362766 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:24.363533 28394 external_mini_cluster.cc:1428] Setting key 32a92b4d5d901ffe7e07ca1d5895b411
I20260809 15:01:24.369288 32515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a: Bootstrap starting.
I20260809 15:01:24.372692 32515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:24.373765 32515 log.cc:824] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:24.376816 32515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a: No bootstrap required, opened a new log
Aug 09 15:01:24 dist-test-slave-q9h9 krb5kdc[32385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:24.385293 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:24.365436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.27.186.190:33467)
0809 15:01:24.366169 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:24.366180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:24.366217 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:01:24.367948 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:24.367968 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:24.368294 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:24.368582 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:24.370322 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:24.371574 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:24.373255 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:24.373901 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:24.374062 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:24.378621 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:24.378628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:24.378641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:24.378677 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:24.381598 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:24.382363 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:24.382369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:24.382374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:24.382540 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:24.382917 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:24.382920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:24.382922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:24.383437 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:24.383646 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:24.384065 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 15:01:24.384377 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":321,"threads_started":1}
I20260809 15:01:24.385337 32515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfbe2b13bf94b368dc43417cd36409a" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33467 } }
I20260809 15:01:24.385752 32515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:24.385867 32515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfbe2b13bf94b368dc43417cd36409a, State: Initialized, Role: FOLLOWER
I20260809 15:01:24.386498 32515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfbe2b13bf94b368dc43417cd36409a" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33467 } }
I20260809 15:01:24.386761 32515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:24.386888 32515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:24.387051 32515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:24.389251 32515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfbe2b13bf94b368dc43417cd36409a" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33467 } }
I20260809 15:01:24.389680 32515 leader_election.cc:302] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cfbe2b13bf94b368dc43417cd36409a; no voters:
I20260809 15:01:24.390226 32515 leader_election.cc:288] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:24.390354 32523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:24.390970 32523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [term 1 LEADER]: Becoming Leader. State: Replica: 6cfbe2b13bf94b368dc43417cd36409a, State: Running, Role: LEADER
I20260809 15:01:24.391506 32523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfbe2b13bf94b368dc43417cd36409a" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33467 } }
I20260809 15:01:24.392215 32515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:24.394140 32525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cfbe2b13bf94b368dc43417cd36409a. Latest consensus state: current_term: 1 leader_uuid: "6cfbe2b13bf94b368dc43417cd36409a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfbe2b13bf94b368dc43417cd36409a" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33467 } } }
I20260809 15:01:24.394112 32524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cfbe2b13bf94b368dc43417cd36409a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfbe2b13bf94b368dc43417cd36409a" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33467 } } }
I20260809 15:01:24.394520 32525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:24.394524 32524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:24.400604 32532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:24.404313 32532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:24.412135 32532 catalog_manager.cc:1347] Generated new cluster ID: 0bf1540341264ef9945192592e696411
I20260809 15:01:24.412235 32532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:24.440678 32532 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:24.442422 32532 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:24.449878 32532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cfbe2b13bf94b368dc43417cd36409a: Generated new TSK 0
I20260809 15:01:24.450763 32532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:24.494691 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33467
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:24.809283 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:24.809533 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:24.809665 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:24.816673 32557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:24.816785 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:24.816926 32557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:24.826090 32557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:24.827152 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:24.828361 32557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:24.840878 32564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:24.840971 32565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:24.840926 32567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:24.841789 32557 server_base.cc:1034] running on GCE node
I20260809 15:01:24.842408 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:24.843261 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:24.844455 32557 hybrid_clock.cc:648] HybridClock initialized: now 1786287684844399 us; error 47 us; skew 500 ppm
I20260809 15:01:24.844750 32557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:24 dist-test-slave-q9h9 krb5kdc[32385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:24.847880 32557 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:24.849303 32557 webserver.cc:460] Webserver started at http://127.27.186.129:43131/ using document root <none> and password file <none>
I20260809 15:01:24.849906 32557 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:24.850059 32557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:24.850358 32557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:24.853015 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "afb6c68517bf443c89bd03dad002d27e"
format_stamp: "Formatted at 2026-08-09 15:01:24 on dist-test-slave-q9h9"
server_key: "cad7d49d55aacf09798e35bfd317738b"
server_key_iv: "1b5594057c802ddcc24ae258194f4b0d"
server_key_version: "encryptionkey@0"
I20260809 15:01:24.853753 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "afb6c68517bf443c89bd03dad002d27e"
format_stamp: "Formatted at 2026-08-09 15:01:24 on dist-test-slave-q9h9"
server_key: "cad7d49d55aacf09798e35bfd317738b"
server_key_iv: "1b5594057c802ddcc24ae258194f4b0d"
server_key_version: "encryptionkey@0"
I20260809 15:01:24.858536 32557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:01:24.861734 32579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:24.862896 32557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:01:24.863037 32557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "afb6c68517bf443c89bd03dad002d27e"
format_stamp: "Formatted at 2026-08-09 15:01:24 on dist-test-slave-q9h9"
server_key: "cad7d49d55aacf09798e35bfd317738b"
server_key_iv: "1b5594057c802ddcc24ae258194f4b0d"
server_key_version: "encryptionkey@0"
I20260809 15:01:24.863225 32557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:24.875418 32557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:24.879177 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:24.879459 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:24.880285 32557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:24.881477 32557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:24.881554 32557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:24.881670 32557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:24.881726 32557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:24.912822 32557 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:39105
I20260809 15:01:24.912891 32766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:39105 every 8 connection(s)
I20260809 15:01:24.913942 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:24.920217 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 32557
I20260809 15:01:24.920403 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:24.921010 28394 external_mini_cluster.cc:1428] Setting key e0fdfeb77f80e52353a41f95f93d59a1
Aug 09 15:01:24 dist-test-slave-q9h9 krb5kdc[32385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:24.933519 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:24.916286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:52231 (local address 127.27.186.190:33467)
0809 15:01:24.916710 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:24.916714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:24.917544 (+ 830us) server_negotiation.cc:407] Connection header received
0809 15:01:24.919558 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:24.919565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:24.919667 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:24.919777 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:24.921368 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:24.922297 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:24.923858 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:24.924278 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:24.924401 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:24.927880 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:24.927885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:24.927889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:24.927923 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:24.930481 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:24.931334 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:24.931338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:24.931341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:24.931467 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:24.931942 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:24.931945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:24.931947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:24.932198 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:24.932308 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:24.933115 (+ 807us) server_negotiation.cc:299] Negotiation successful
0809 15:01:24.933269 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":219,"threads_started":1}
I20260809 15:01:24.934918 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:24.916661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33467 (local address 127.27.186.129:52231)
0809 15:01:24.917392 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:24.917440 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:24.919285 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:24.920038 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:24.920046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:24.920597 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:24.921184 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:24.921204 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:24.922456 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:24.922460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:24.923706 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:24.923719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:24.924504 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:24.924512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:24.924747 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:24.925368 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:24.925391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:24.927662 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:24.930637 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:24.930646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:24.930659 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:24.931212 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:24.931586 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:24.931592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:24.931596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:24.931839 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:24.932325 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:24.932333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:24.932759 (+ 426us) client_negotiation.cc:769] Sending connection context
0809 15:01:24.933074 (+ 315us) client_negotiation.cc:240] Negotiation successful
0809 15:01:24.933552 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":270,"threads_started":1}
I20260809 15:01:24.936239 32767 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33467
I20260809 15:01:24.936560 32767 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:24.937443 32767 heartbeater.cc:511] Master 127.27.186.190:33467 requested a full tablet report, sending...
I20260809 15:01:24.939558 32450 ts_manager.cc:194] Registered new tserver with Master: afb6c68517bf443c89bd03dad002d27e (127.27.186.129:39105)
I20260809 15:01:24.940965 32450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:52231
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:24.962267 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33467
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:25.277417 309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:25.277689 309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:25.277819 309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:25.284878 309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:25.285002 309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:25.285171 309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:25.294632 309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:25.295814 309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:25.297128 309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:25.309578 318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:25.309712 320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:25.309741 322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:25.310679 309 server_base.cc:1034] running on GCE node
I20260809 15:01:25.311216 309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:25.312045 309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:25.313221 309 hybrid_clock.cc:648] HybridClock initialized: now 1786287685313179 us; error 61 us; skew 500 ppm
I20260809 15:01:25.313490 309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:25 dist-test-slave-q9h9 krb5kdc[32385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:25.316398 309 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:25.317840 309 webserver.cc:460] Webserver started at http://127.27.186.130:37621/ using document root <none> and password file <none>
I20260809 15:01:25.318611 309 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:25.318750 309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:25.319037 309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:25.322279 309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "f5f95015494a4d7b84068c634d14e269"
format_stamp: "Formatted at 2026-08-09 15:01:25 on dist-test-slave-q9h9"
server_key: "7f89f55a853ec95e69a2fd1d7f941e5a"
server_key_iv: "cc9fb6813b6a0520d1566e1d8501f8cc"
server_key_version: "encryptionkey@0"
I20260809 15:01:25.323024 309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "f5f95015494a4d7b84068c634d14e269"
format_stamp: "Formatted at 2026-08-09 15:01:25 on dist-test-slave-q9h9"
server_key: "7f89f55a853ec95e69a2fd1d7f941e5a"
server_key_iv: "cc9fb6813b6a0520d1566e1d8501f8cc"
server_key_version: "encryptionkey@0"
I20260809 15:01:25.327746 309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:25.330926 334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:25.332046 309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:01:25.332191 309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "f5f95015494a4d7b84068c634d14e269"
format_stamp: "Formatted at 2026-08-09 15:01:25 on dist-test-slave-q9h9"
server_key: "7f89f55a853ec95e69a2fd1d7f941e5a"
server_key_iv: "cc9fb6813b6a0520d1566e1d8501f8cc"
server_key_version: "encryptionkey@0"
I20260809 15:01:25.332356 309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:25.348060 309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:25.351635 309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:25.351871 309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:25.352686 309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:25.353979 309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:25.354079 309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:25.354205 309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:25.354254 309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:25.387578 309 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:41587
I20260809 15:01:25.387633 517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:41587 every 8 connection(s)
I20260809 15:01:25.388670 309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:25.398777 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 309
I20260809 15:01:25.398960 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:25.399521 28394 external_mini_cluster.cc:1428] Setting key 55a3df70af14e3744388d73755be3470
Aug 09 15:01:25 dist-test-slave-q9h9 krb5kdc[32385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:25.407182 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:25.390838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:46609 (local address 127.27.186.190:33467)
0809 15:01:25.390984 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:25.390988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:25.391998 (+ 1010us) server_negotiation.cc:407] Connection header received
0809 15:01:25.393787 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:25.393793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:25.393874 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:25.393961 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:25.395385 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.396141 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.397627 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.398046 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.398163 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:25.402070 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:25.402075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:25.402079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:25.402107 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:25.404445 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:25.405092 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:25.405097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:25.405100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:25.405228 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:25.405612 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:25.405616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:25.405620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:25.405867 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:25.406007 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:25.406642 (+ 635us) server_negotiation.cc:299] Negotiation successful
0809 15:01:25.406868 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:01:25.407976 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:25.391202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33467 (local address 127.27.186.130:46609)
0809 15:01:25.391849 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:25.391890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:25.393532 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:25.394217 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:25.394226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:25.394679 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:25.395222 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:25.395241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.396301 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.396308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:25.397493 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:25.397506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.398267 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.398276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:25.398530 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:25.399180 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:25.399223 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:25.401845 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:25.404574 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:25.404583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:25.404596 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:25.404980 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:25.405331 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:25.405336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:25.405340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:25.405508 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:25.405987 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:25.405995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:25.406314 (+ 319us) client_negotiation.cc:769] Sending connection context
0809 15:01:25.406534 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 15:01:25.406988 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
I20260809 15:01:25.409241 518 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33467
I20260809 15:01:25.409495 518 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:25.410182 518 heartbeater.cc:511] Master 127.27.186.190:33467 requested a full tablet report, sending...
I20260809 15:01:25.411505 32450 ts_manager.cc:194] Registered new tserver with Master: f5f95015494a4d7b84068c634d14e269 (127.27.186.130:41587)
I20260809 15:01:25.412293 32450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:46609
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:25.439072 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33467
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:25.745723 528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:25.745955 528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:25.746066 528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:25.752883 528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:25.752977 528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:25.753087 528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:25.762063 528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:25.763068 528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:25.764219 528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:25.776124 542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:25.776227 540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:25.776082 544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:25.777068 528 server_base.cc:1034] running on GCE node
I20260809 15:01:25.777585 528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:25.778475 528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:25.779670 528 hybrid_clock.cc:648] HybridClock initialized: now 1786287685779630 us; error 27 us; skew 500 ppm
I20260809 15:01:25.779950 528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:25 dist-test-slave-q9h9 krb5kdc[32385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:25.782809 528 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:25.784042 528 webserver.cc:460] Webserver started at http://127.27.186.131:34629/ using document root <none> and password file <none>
I20260809 15:01:25.784641 528 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:25.784765 528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:25.785041 528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:25.787681 528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "8ec88085f8554363ac90d1cdba3f59f4"
format_stamp: "Formatted at 2026-08-09 15:01:25 on dist-test-slave-q9h9"
server_key: "9f9c7d6c4175283a113f867056499481"
server_key_iv: "900eb51225a332e6ab21ac4aeb4b693f"
server_key_version: "encryptionkey@0"
I20260809 15:01:25.788422 528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "8ec88085f8554363ac90d1cdba3f59f4"
format_stamp: "Formatted at 2026-08-09 15:01:25 on dist-test-slave-q9h9"
server_key: "9f9c7d6c4175283a113f867056499481"
server_key_iv: "900eb51225a332e6ab21ac4aeb4b693f"
server_key_version: "encryptionkey@0"
I20260809 15:01:25.793128 528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:25.796403 555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:25.797561 528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:25.797700 528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "8ec88085f8554363ac90d1cdba3f59f4"
format_stamp: "Formatted at 2026-08-09 15:01:25 on dist-test-slave-q9h9"
server_key: "9f9c7d6c4175283a113f867056499481"
server_key_iv: "900eb51225a332e6ab21ac4aeb4b693f"
server_key_version: "encryptionkey@0"
I20260809 15:01:25.797868 528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:25.810304 528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:25.813727 528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:25.813978 528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:25.814872 528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:25.816088 528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:25.816165 528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:25.816285 528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:25.816345 528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:25.845535 528 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:46849
I20260809 15:01:25.845577 727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:46849 every 8 connection(s)
I20260809 15:01:25.846704 528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:25.853356 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 528
I20260809 15:01:25.853524 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:25.853935 28394 external_mini_cluster.cc:1428] Setting key b5b657466b5f02103b15ac5a7c63beab
Aug 09 15:01:25 dist-test-slave-q9h9 krb5kdc[32385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:25.864553 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:25.848931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:58101 (local address 127.27.186.190:33467)
0809 15:01:25.849098 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:25.849103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:25.850204 (+ 1101us) server_negotiation.cc:407] Connection header received
0809 15:01:25.851914 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:25.851921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:25.852000 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:25.852088 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:25.853438 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.854261 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.856012 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.856485 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.856588 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:25.859652 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:25.859657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:25.859661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:25.859693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:25.861840 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:25.862504 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:25.862510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:25.862513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:25.862633 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:25.863022 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:25.863026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:25.863028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:25.863290 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:25.863419 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:25.864022 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 15:01:25.864232 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 15:01:25.865324 731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:25.849363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33467 (local address 127.27.186.131:58101)
0809 15:01:25.850050 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:25.850094 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:25.851671 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:25.852315 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:25.852322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:25.852766 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:25.853278 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:25.853296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.854402 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.854406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:25.855886 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:25.855899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:25.856688 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:25.856696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:25.856927 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:25.857447 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:25.857466 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:25.859457 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:25.861969 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:25.861977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:25.861990 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:25.862398 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:25.862731 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:25.862736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:25.862740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:25.862912 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:25.863410 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:25.863417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:25.863691 (+ 274us) client_negotiation.cc:769] Sending connection context
0809 15:01:25.863893 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 15:01:25.864335 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260809 15:01:25.866532 729 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33467
I20260809 15:01:25.866763 729 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:25.867460 729 heartbeater.cc:511] Master 127.27.186.190:33467 requested a full tablet report, sending...
I20260809 15:01:25.868777 32450 ts_manager.cc:194] Registered new tserver with Master: 8ec88085f8554363ac90d1cdba3f59f4 (127.27.186.131:46849)
I20260809 15:01:25.869509 32450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:58101
I20260809 15:01:25.880251 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 15:01:25.910411 301 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:25.908225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.27.186.190:33467)
0809 15:01:25.908432 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:25.908439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:25.908925 (+ 486us) server_negotiation.cc:407] Connection header received
0809 15:01:25.909060 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:25.909064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:25.909124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:25.909194 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:25.910211 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51182: BlockingRecv error: recv got EOF from 127.0.0.1:51182 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:01:25.914685 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32557
W20260809 15:01:25.917676 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32557: Not authorized: Client connection negotiation failed: client connection to 127.27.186.129:39105: server requires authentication, but client does not have Kerberos credentials available
I20260809 15:01:25.917829 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 32557
I20260809 15:01:25.939038 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 309
W20260809 15:01:25.941417 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 309: Not authorized: Client connection negotiation failed: client connection to 127.27.186.130:41587: server requires authentication, but client does not have Kerberos credentials available
I20260809 15:01:25.941582 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 309
I20260809 15:01:25.962033 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 528
W20260809 15:01:25.964681 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 528: Not authorized: Client connection negotiation failed: client connection to 127.27.186.131:46849: server requires authentication, but client does not have Kerberos credentials available
I20260809 15:01:25.964792 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 528
I20260809 15:01:25.986006 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32400
I20260809 15:01:25.987097 32503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:26.085948 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 32400
2026-08-09T15:01:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3487 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:26 dist-test-slave-q9h9 krb5kdc[752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:26 dist-test-slave-q9h9 krb5kdc[752](info): set up 2 sockets
Aug 09 15:01:26 dist-test-slave-q9h9 krb5kdc[752](info): commencing operation
krb5kdc: starting...
W20260809 15:01:27.276096 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:27 dist-test-slave-q9h9 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:27.447086 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:46275
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:46275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:27.746380 772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:27.746627 772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:27.746758 772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:27.753768 772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:27.753877 772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:27.753961 772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:27.754062 772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:27.754134 772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:27.763794 772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:46275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:46275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:27.764922 772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:27.766157 772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:27.777002 778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:27.777035 779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:27.776957 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:27.777926 772 server_base.cc:1034] running on GCE node
I20260809 15:01:27.778604 772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:27.779824 772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:27.781011 772 hybrid_clock.cc:648] HybridClock initialized: now 1786287687780968 us; error 35 us; skew 500 ppm
I20260809 15:01:27.781323 772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:27 dist-test-slave-q9h9 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:27.788496 772 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:27.789803 772 webserver.cc:460] Webserver started at http://127.27.186.190:46653/ using document root <none> and password file <none>
I20260809 15:01:27.790444 772 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:27.790561 772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:27.790834 772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:27.793509 772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "dcdebe9824a9443699ab9dc8c38a54df"
format_stamp: "Formatted at 2026-08-09 15:01:27 on dist-test-slave-q9h9"
server_key: "a89de4622e7f018ceb19963ae4d752a3"
server_key_iv: "9853be8f1f6659a874e6ef842f44d4fe"
server_key_version: "encryptionkey@0"
I20260809 15:01:27.794308 772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "dcdebe9824a9443699ab9dc8c38a54df"
format_stamp: "Formatted at 2026-08-09 15:01:27 on dist-test-slave-q9h9"
server_key: "a89de4622e7f018ceb19963ae4d752a3"
server_key_iv: "9853be8f1f6659a874e6ef842f44d4fe"
server_key_version: "encryptionkey@0"
I20260809 15:01:27.798939 772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:01:27.802037 793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:27.803176 772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:01:27.803334 772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "dcdebe9824a9443699ab9dc8c38a54df"
format_stamp: "Formatted at 2026-08-09 15:01:27 on dist-test-slave-q9h9"
server_key: "a89de4622e7f018ceb19963ae4d752a3"
server_key_iv: "9853be8f1f6659a874e6ef842f44d4fe"
server_key_version: "encryptionkey@0"
I20260809 15:01:27.803490 772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:27.857378 772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:27.861233 772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:27.861490 772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:27.877336 772 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:46275
I20260809 15:01:27.877369 870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:46275 every 8 connection(s)
I20260809 15:01:27.878515 772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:27.880402 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 772
I20260809 15:01:27.880602 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:27.881217 28394 external_mini_cluster.cc:1428] Setting key 82b7ce4804552ba6c133bc10cefd7889
I20260809 15:01:27.881856 871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:27.893170 871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df: Bootstrap starting.
Aug 09 15:01:27 dist-test-slave-q9h9 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:27.896317 871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:27.897292 871 log.cc:824] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:27.899696 871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df: No bootstrap required, opened a new log
I20260809 15:01:27.900938 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:27.882896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.27.186.190:46275)
0809 15:01:27.883626 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:27.883637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:27.883673 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:01:27.885127 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:27.885146 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:27.885500 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:27.885838 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:27.887299 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:27.888573 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:27.889773 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:27.890458 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:27.890619 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:27.894473 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:27.894481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:27.894495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:27.894537 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:27.897264 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:27.897908 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:27.897914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:27.897919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:27.898131 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:27.898492 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:27.898496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:27.898498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:27.898961 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:27.899227 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:27.899658 (+ 431us) server_negotiation.cc:299] Negotiation successful
0809 15:01:27.900052 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":338,"threads_started":1}
I20260809 15:01:27.904403 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:01:27.908388 871 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdebe9824a9443699ab9dc8c38a54df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46275 } }
I20260809 15:01:27.908761 871 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:27.908859 871 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdebe9824a9443699ab9dc8c38a54df, State: Initialized, Role: FOLLOWER
I20260809 15:01:27.909407 871 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdebe9824a9443699ab9dc8c38a54df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46275 } }
I20260809 15:01:27.909657 871 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:27.909789 871 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:27.909947 871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:27.912572 871 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdebe9824a9443699ab9dc8c38a54df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46275 } }
I20260809 15:01:27.913090 871 leader_election.cc:302] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcdebe9824a9443699ab9dc8c38a54df; no voters:
I20260809 15:01:27.913632 871 leader_election.cc:288] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:27.913712 881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:27.915526 871 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:27.920634 881 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [term 1 LEADER]: Becoming Leader. State: Replica: dcdebe9824a9443699ab9dc8c38a54df, State: Running, Role: LEADER
I20260809 15:01:27.921054 881 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdebe9824a9443699ab9dc8c38a54df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46275 } }
I20260809 15:01:27.923317 883 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcdebe9824a9443699ab9dc8c38a54df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdebe9824a9443699ab9dc8c38a54df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46275 } } }
I20260809 15:01:27.923671 883 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:27.924485 886 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdebe9824a9443699ab9dc8c38a54df. Latest consensus state: current_term: 1 leader_uuid: "dcdebe9824a9443699ab9dc8c38a54df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdebe9824a9443699ab9dc8c38a54df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46275 } } }
I20260809 15:01:27.925189 886 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:27.930933 895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:27.935705 895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:27.944470 895 catalog_manager.cc:1347] Generated new cluster ID: c16972b635b7497ca09e0c72ade61f3f
I20260809 15:01:27.944612 895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:27.953537 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:27.945154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.27.186.190:46275)
0809 15:01:27.945329 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:27.945335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:27.945361 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:01:27.945601 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:27.945607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:27.945694 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:27.945769 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:27.947079 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:27.948133 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:27.949882 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:27.950540 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:27.950638 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:27.951927 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:27.951933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:27.951939 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:27.952722 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:27.952788 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:27.952822 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:27.952893 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:27.953098 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 15:01:27.953183 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:01:27.965238 895 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:27.966895 895 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:27.973857 895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcdebe9824a9443699ab9dc8c38a54df: Generated new TSK 0
I20260809 15:01:27.974745 895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:01:27.981205 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 772
I20260809 15:01:27.982414 862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:28.067729 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 772
2026-08-09T15:01:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1978 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:28 dist-test-slave-q9h9 krb5kdc[927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:28 dist-test-slave-q9h9 krb5kdc[927](info): set up 2 sockets
Aug 09 15:01:28 dist-test-slave-q9h9 krb5kdc[927](info): commencing operation
krb5kdc: starting...
W20260809 15:01:29.247408 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:29 dist-test-slave-q9h9 krb5kdc[927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:29.421155 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43219
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:29.722651 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:29.722900 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:29.723029 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:29.729697 945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:29.729794 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:29.729869 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:29.729949 945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:29.730010 945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:29.738919 945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:29.739970 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:29.741081 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:29.751463 952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:29.751518 953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:29.751431 957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:29.752297 945 server_base.cc:1034] running on GCE node
I20260809 15:01:29.753031 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:29.754290 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:29.755501 945 hybrid_clock.cc:648] HybridClock initialized: now 1786287689755446 us; error 43 us; skew 500 ppm
I20260809 15:01:29.755823 945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:29 dist-test-slave-q9h9 krb5kdc[927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287689, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:29.758761 945 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:29.760128 945 webserver.cc:460] Webserver started at http://127.27.186.190:36229/ using document root <none> and password file <none>
I20260809 15:01:29.760747 945 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:29.760864 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:29.761142 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:29.763959 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "e03bea6d87c248fd9d5814e7efaa55a2"
format_stamp: "Formatted at 2026-08-09 15:01:29 on dist-test-slave-q9h9"
server_key: "8a8e0a669afc264e6bd9020ba7875f6b"
server_key_iv: "ead30c65c54a5b4902d2506fec120f92"
server_key_version: "encryptionkey@0"
I20260809 15:01:29.764694 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "e03bea6d87c248fd9d5814e7efaa55a2"
format_stamp: "Formatted at 2026-08-09 15:01:29 on dist-test-slave-q9h9"
server_key: "8a8e0a669afc264e6bd9020ba7875f6b"
server_key_iv: "ead30c65c54a5b4902d2506fec120f92"
server_key_version: "encryptionkey@0"
I20260809 15:01:29.769405 945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:29.772744 966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:29.773932 945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:29.774098 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "e03bea6d87c248fd9d5814e7efaa55a2"
format_stamp: "Formatted at 2026-08-09 15:01:29 on dist-test-slave-q9h9"
server_key: "8a8e0a669afc264e6bd9020ba7875f6b"
server_key_iv: "ead30c65c54a5b4902d2506fec120f92"
server_key_version: "encryptionkey@0"
I20260809 15:01:29.774271 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:29.796990 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:29.800457 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:29.800671 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:29.816778 945 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:43219
I20260809 15:01:29.816831 1040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:43219 every 8 connection(s)
I20260809 15:01:29.817927 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:29.821238 1041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:29.824199 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 945
I20260809 15:01:29.824393 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:29.824981 28394 external_mini_cluster.cc:1428] Setting key a0a4204cb0d60c6441f328218dad7541
I20260809 15:01:29.832688 1041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2: Bootstrap starting.
I20260809 15:01:29.836081 1041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:29.837303 1041 log.cc:824] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:01:29 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:29.840186 1041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2: No bootstrap required, opened a new log
I20260809 15:01:29.846321 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:29.826758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.27.186.190:43219)
0809 15:01:29.827448 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:29.827459 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:29.827495 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:01:29.828977 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:29.828999 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:29.829342 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:29.829652 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:29.831254 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:29.832453 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:29.834159 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:29.834794 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:29.834925 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:29.839601 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:29.839612 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:29.839629 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:29.839678 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:29.842658 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:29.843423 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:29.843428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:29.843437 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:29.843594 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:29.843946 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:29.843949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:29.843951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:29.844417 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:29.844644 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:29.845035 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 15:01:29.845332 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":324,"threads_started":1}
I20260809 15:01:29.848198 1041 raft_consensus.cc:348] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43219 } }
I20260809 15:01:29.848516 1041 raft_consensus.cc:374] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:29.848610 1041 raft_consensus.cc:729] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03bea6d87c248fd9d5814e7efaa55a2, State: Initialized, Role: FOLLOWER
I20260809 15:01:29.849105 1041 consensus_queue.cc:260] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43219 } }
I20260809 15:01:29.849315 1041 raft_consensus.cc:388] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:29.849442 1041 raft_consensus.cc:482] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:29.849583 1041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:29.851588 1041 raft_consensus.cc:504] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43219 } }
I20260809 15:01:29.852041 1041 leader_election.cc:302] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e03bea6d87c248fd9d5814e7efaa55a2; no voters:
I20260809 15:01:29.852522 1041 leader_election.cc:288] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:29.852627 1051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:29.853241 1051 raft_consensus.cc:686] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [term 1 LEADER]: Becoming Leader. State: Replica: e03bea6d87c248fd9d5814e7efaa55a2, State: Running, Role: LEADER
I20260809 15:01:29.853798 1051 consensus_queue.cc:237] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43219 } }
I20260809 15:01:29.854296 1041 sys_catalog.cc:564] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:29.856633 1053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43219 } } }
I20260809 15:01:29.856993 1053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:29.857656 1054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e03bea6d87c248fd9d5814e7efaa55a2. Latest consensus state: current_term: 1 leader_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03bea6d87c248fd9d5814e7efaa55a2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43219 } } }
I20260809 15:01:29.858067 1054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:29.858381 1059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:29.863570 1059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:29.870786 1059 catalog_manager.cc:1347] Generated new cluster ID: f28e36ac3c4449dfb04ebd71e2d91208
I20260809 15:01:29.870886 1059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:29.878976 1059 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:29.880043 1059 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:29.886404 1059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e03bea6d87c248fd9d5814e7efaa55a2: Generated new TSK 0
I20260809 15:01:29.887024 1059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:29.935237 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43219
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:30.253290 1076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:30.253554 1076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:30.253693 1076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:30.260860 1076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:30.260972 1076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:30.261117 1076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:30.270350 1076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:30.271456 1076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:30.272661 1076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:30.284700 1086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:30.284674 1087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:30.284783 1093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:30.285558 1076 server_base.cc:1034] running on GCE node
I20260809 15:01:30.286147 1076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:30.286974 1076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:30.288196 1076 hybrid_clock.cc:648] HybridClock initialized: now 1786287690288139 us; error 40 us; skew 500 ppm
I20260809 15:01:30.288513 1076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:30 dist-test-slave-q9h9 krb5kdc[927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:30.291630 1076 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:30.292945 1076 webserver.cc:460] Webserver started at http://127.27.186.129:37033/ using document root <none> and password file <none>
I20260809 15:01:30.293556 1076 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:30.293673 1076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:30.293946 1076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:30.296648 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "5f48f74365914f6a93708dd95329456f"
format_stamp: "Formatted at 2026-08-09 15:01:30 on dist-test-slave-q9h9"
server_key: "3fefb6c4471e39186fdb76d6364ee18d"
server_key_iv: "3e2bea0ac6ac6b944241ee08223244e8"
server_key_version: "encryptionkey@0"
I20260809 15:01:30.297387 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "5f48f74365914f6a93708dd95329456f"
format_stamp: "Formatted at 2026-08-09 15:01:30 on dist-test-slave-q9h9"
server_key: "3fefb6c4471e39186fdb76d6364ee18d"
server_key_iv: "3e2bea0ac6ac6b944241ee08223244e8"
server_key_version: "encryptionkey@0"
I20260809 15:01:30.302081 1076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:01:30.305469 1100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:30.306655 1076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:30.306807 1076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "5f48f74365914f6a93708dd95329456f"
format_stamp: "Formatted at 2026-08-09 15:01:30 on dist-test-slave-q9h9"
server_key: "3fefb6c4471e39186fdb76d6364ee18d"
server_key_iv: "3e2bea0ac6ac6b944241ee08223244e8"
server_key_version: "encryptionkey@0"
I20260809 15:01:30.306969 1076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:30.329135 1076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:30.332612 1076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:30.332885 1076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:30.333704 1076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:30.334992 1076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:30.335076 1076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:30.335214 1076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:30.335265 1076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:30.365556 1076 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:43475
I20260809 15:01:30.365626 1280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:43475 every 8 connection(s)
I20260809 15:01:30.366724 1076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:30.371727 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 1076
I20260809 15:01:30.371948 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:30.372850 28394 external_mini_cluster.cc:1428] Setting key 15c59cee6d34133245f15cfc1c64cba7
Aug 09 15:01:30 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:30.388255 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:30.368986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:50091 (local address 127.27.186.190:43219)
0809 15:01:30.369434 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:30.369438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:30.370227 (+ 789us) server_negotiation.cc:407] Connection header received
0809 15:01:30.372453 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:30.372458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:30.372527 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:30.372616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:30.374403 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.375307 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.377634 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.378071 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.378165 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:30.382162 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:30.382167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:30.382170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:30.382202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:30.385164 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:30.385863 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:30.385869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:30.385872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:30.386055 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:30.386544 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:30.386548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:30.386551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:30.386924 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:30.387061 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:30.387716 (+ 655us) server_negotiation.cc:299] Negotiation successful
0809 15:01:30.387916 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":249,"threads_started":1}
I20260809 15:01:30.389056 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:30.369349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43219 (local address 127.27.186.129:50091)
0809 15:01:30.370087 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:30.370125 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:30.372155 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:30.372888 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:30.372900 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:30.373622 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:30.374233 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:30.374253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.375479 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.375485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:30.377496 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:30.377513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.378292 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.378300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:30.378624 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:30.379308 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:30.379338 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:30.381906 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:30.385308 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:30.385315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:30.385324 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:30.385735 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:30.386183 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:30.386188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:30.386192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:30.386399 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:30.387047 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:30.387052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:30.387347 (+ 295us) client_negotiation.cc:769] Sending connection context
0809 15:01:30.387583 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 15:01:30.387928 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":274,"threads_started":1}
I20260809 15:01:30.390465 1284 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43219
I20260809 15:01:30.390753 1284 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:30.391448 1284 heartbeater.cc:511] Master 127.27.186.190:43219 requested a full tablet report, sending...
I20260809 15:01:30.393452 992 ts_manager.cc:194] Registered new tserver with Master: 5f48f74365914f6a93708dd95329456f (127.27.186.129:43475)
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
I20260809 15:01:30.394948 992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:50091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:30.415200 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43219
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:30.735412 1295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:30.735690 1295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:30.735837 1295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:30.743018 1295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:30.743135 1295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:30.743294 1295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:30.752676 1295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:30.753752 1295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:30.755025 1295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:30.767625 1307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:30.767604 1305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:30.767678 1302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:30.768442 1295 server_base.cc:1034] running on GCE node
I20260809 15:01:30.769007 1295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:30.769874 1295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:30.771090 1295 hybrid_clock.cc:648] HybridClock initialized: now 1786287690771026 us; error 37 us; skew 500 ppm
I20260809 15:01:30.771377 1295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:30 dist-test-slave-q9h9 krb5kdc[927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:30.774366 1295 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:30.775691 1295 webserver.cc:460] Webserver started at http://127.27.186.130:45123/ using document root <none> and password file <none>
I20260809 15:01:30.776312 1295 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:30.776441 1295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:30.776741 1295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:30.779608 1295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "a47b997518c444dea6f3677d91984ad5"
format_stamp: "Formatted at 2026-08-09 15:01:30 on dist-test-slave-q9h9"
server_key: "da02d8a8a412872a8e6ee9f7cdb0be85"
server_key_iv: "099dc69cfa574f2cce4e969c3506f44e"
server_key_version: "encryptionkey@0"
I20260809 15:01:30.780405 1295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "a47b997518c444dea6f3677d91984ad5"
format_stamp: "Formatted at 2026-08-09 15:01:30 on dist-test-slave-q9h9"
server_key: "da02d8a8a412872a8e6ee9f7cdb0be85"
server_key_iv: "099dc69cfa574f2cce4e969c3506f44e"
server_key_version: "encryptionkey@0"
I20260809 15:01:30.785382 1295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 15:01:30.788802 1320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:30.789956 1295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:30.790153 1295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "a47b997518c444dea6f3677d91984ad5"
format_stamp: "Formatted at 2026-08-09 15:01:30 on dist-test-slave-q9h9"
server_key: "da02d8a8a412872a8e6ee9f7cdb0be85"
server_key_iv: "099dc69cfa574f2cce4e969c3506f44e"
server_key_version: "encryptionkey@0"
I20260809 15:01:30.790349 1295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:30.813925 1295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:30.818188 1295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:30.818532 1295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:30.819607 1295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:30.821241 1295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:30.821338 1295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:30.821458 1295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:30.821524 1295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:30.857116 1295 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33653
I20260809 15:01:30.857163 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33653 every 8 connection(s)
I20260809 15:01:30.858364 1295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:30.859786 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 1295
I20260809 15:01:30.859941 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:30.860550 28394 external_mini_cluster.cc:1428] Setting key f028f2828e38ad00a444c3dde79a94af
Aug 09 15:01:30 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:30.878894 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:30.860808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:40379 (local address 127.27.186.190:43219)
0809 15:01:30.861010 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:30.861017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:30.862359 (+ 1342us) server_negotiation.cc:407] Connection header received
0809 15:01:30.864796 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:30.864799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:30.864859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:30.864933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:30.866812 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.867506 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.869574 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.869984 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.870107 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:30.873483 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:30.873488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:30.873491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:30.873523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:30.875966 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:30.876616 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:30.876620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:30.876622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:30.876750 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:30.877179 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:30.877183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:30.877185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:30.877444 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:30.877584 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:30.878331 (+ 747us) server_negotiation.cc:299] Negotiation successful
0809 15:01:30.878565 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:01:30.879737 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:30.861226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43219 (local address 127.27.186.130:40379)
0809 15:01:30.862182 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:30.862241 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:30.864493 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:30.865194 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:30.865204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:30.865871 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:30.866648 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:30.866676 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.867668 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.867674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:30.869417 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:30.869434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:30.870217 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:30.870225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:30.870498 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:30.871196 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:30.871219 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:30.873289 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:30.876092 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:30.876100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:30.876114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:30.876507 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:30.876859 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:30.876864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:30.876868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:30.877077 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:30.877627 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:30.877635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:30.877905 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 15:01:30.878213 (+ 308us) client_negotiation.cc:240] Negotiation successful
0809 15:01:30.878730 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":375,"threads_started":1}
I20260809 15:01:30.880985 1493 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43219
I20260809 15:01:30.881242 1493 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
I20260809 15:01:30.881906 1493 heartbeater.cc:511] Master 127.27.186.190:43219 requested a full tablet report, sending...
I20260809 15:01:30.883212 992 ts_manager.cc:194] Registered new tserver with Master: a47b997518c444dea6f3677d91984ad5 (127.27.186.130:33653)
I20260809 15:01:30.883958 992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:40379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:30.903023 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43219
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:31.208141 1499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:31.208401 1499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:31.208496 1499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:31.215420 1499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:31.215513 1499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:31.215626 1499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:31.224628 1499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:31.225695 1499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:31.226913 1499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:31.238983 1505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:31.239105 1506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:31.239423 1508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:31.240007 1499 server_base.cc:1034] running on GCE node
I20260809 15:01:31.240566 1499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:31.241371 1499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:31.242560 1499 hybrid_clock.cc:648] HybridClock initialized: now 1786287691242508 us; error 46 us; skew 500 ppm
I20260809 15:01:31.242854 1499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:31.245772 1499 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:31.247148 1499 webserver.cc:460] Webserver started at http://127.27.186.131:42769/ using document root <none> and password file <none>
I20260809 15:01:31.247771 1499 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:31.247888 1499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:31.248158 1499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:31.250914 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "c23ee1ffc8ab4f65a8f529263947c0bf"
format_stamp: "Formatted at 2026-08-09 15:01:31 on dist-test-slave-q9h9"
server_key: "ae428c4e6e77654142e96e04e0d44518"
server_key_iv: "e093340d79372b8bc5b8e490e9a7345c"
server_key_version: "encryptionkey@0"
I20260809 15:01:31.251637 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "c23ee1ffc8ab4f65a8f529263947c0bf"
format_stamp: "Formatted at 2026-08-09 15:01:31 on dist-test-slave-q9h9"
server_key: "ae428c4e6e77654142e96e04e0d44518"
server_key_iv: "e093340d79372b8bc5b8e490e9a7345c"
server_key_version: "encryptionkey@0"
I20260809 15:01:31.256384 1499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:31.259654 1519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:31.260839 1499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 15:01:31.260987 1499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "c23ee1ffc8ab4f65a8f529263947c0bf"
format_stamp: "Formatted at 2026-08-09 15:01:31 on dist-test-slave-q9h9"
server_key: "ae428c4e6e77654142e96e04e0d44518"
server_key_iv: "e093340d79372b8bc5b8e490e9a7345c"
server_key_version: "encryptionkey@0"
I20260809 15:01:31.261152 1499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:31.271644 1499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:31.275362 1499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:31.275614 1499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:31.276434 1499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:31.277676 1499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:31.277751 1499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:31.277863 1499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:31.277917 1499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:31.307200 1499 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:44257
I20260809 15:01:31.307265 1685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:44257 every 8 connection(s)
I20260809 15:01:31.308367 1499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:31.317482 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 1499
I20260809 15:01:31.317628 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:31.318100 28394 external_mini_cluster.cc:1428] Setting key 8468a664445d4f6b68c3442ecafe6f32
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:31.326426 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:31.310760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:54233 (local address 127.27.186.190:43219)
0809 15:01:31.310952 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:31.310959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:31.312094 (+ 1135us) server_negotiation.cc:407] Connection header received
0809 15:01:31.313828 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:31.313834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:31.313916 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:31.314001 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:31.315437 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.316190 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.317700 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.318182 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.318304 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:31.321724 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:31.321728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:31.321731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:31.321763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:31.324055 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.324666 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.324669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.324672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.324779 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.325094 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.325097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.325099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.325372 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:31.325478 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:31.325996 (+ 518us) server_negotiation.cc:299] Negotiation successful
0809 15:01:31.326195 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:01:31.327217 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:31.311261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43219 (local address 127.27.186.131:54233)
0809 15:01:31.311946 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:31.311981 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:31.313563 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:31.314251 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:31.314259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:31.314733 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:31.315268 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:31.315285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.316340 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.316344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:31.317507 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:31.317525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.318384 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.318392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:31.318625 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:31.319306 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:31.319336 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:31.321544 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:31.324168 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:31.324175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:31.324185 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:31.324577 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:31.324851 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:31.324853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:31.324856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:31.325021 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:31.325461 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:31.325466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:31.325730 (+ 264us) client_negotiation.cc:769] Sending connection context
0809 15:01:31.325939 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 15:01:31.326311 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":260,"threads_started":1}
I20260809 15:01:31.328341 1686 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43219
I20260809 15:01:31.328583 1686 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:31.329191 1686 heartbeater.cc:511] Master 127.27.186.190:43219 requested a full tablet report, sending...
I20260809 15:01:31.330435 992 ts_manager.cc:194] Registered new tserver with Master: c23ee1ffc8ab4f65a8f529263947c0bf (127.27.186.131:44257)
I20260809 15:01:31.331203 992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:54233
I20260809 15:01:31.332979 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:31.382071 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:31.368485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.27.186.190:43219)
0809 15:01:31.368642 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:31.368646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:31.369112 (+ 466us) server_negotiation.cc:407] Connection header received
0809 15:01:31.369284 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:31.369288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:31.369352 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:31.369435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:31.370826 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.371536 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.372757 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.373158 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.373266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:31.377048 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:31.377053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:31.377056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:31.377088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:31.379584 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.380153 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.380157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.380159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.380283 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.380692 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.380697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.380699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.380999 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:31.381224 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:31.381535 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 15:01:31.381750 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 15:01:31.382556 992 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46100
I20260809 15:01:31.384114 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1076
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:31.397789 1284 heartbeater.cc:503] Master 127.27.186.190:43219 was elected leader, sending a full tablet report...
I20260809 15:01:31.398337 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:31.384986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.27.186.129:43475)
0809 15:01:31.385554 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:31.385560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:31.385585 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:31.385634 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:31.385640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:31.385814 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:31.385925 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:31.387024 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.387950 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.389124 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.389657 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.389749 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:31.393135 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:31.393145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:31.393158 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:31.393211 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:31.395903 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.396507 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.396514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.396516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.396643 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.396950 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.396954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.396955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.397267 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:31.397479 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:31.397723 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 15:01:31.397933 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":300,"threads_started":1}
W20260809 15:01:31.399458 1271 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33588
W20260809 15:01:31.400453 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1076: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:31.400642 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 1076
I20260809 15:01:31.444177 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1295
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:31.458585 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:31.445019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.27.186.130:33653)
0809 15:01:31.445570 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:31.445577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:31.445602 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:31.445660 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:31.445667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:31.445869 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:31.445991 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:31.447104 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.448114 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.449233 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.449738 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.449835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:31.453491 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:31.453502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:31.453515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:31.453549 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:31.456344 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.456854 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.456860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.456862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.456973 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.457236 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.457238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.457240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.457539 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:31.457768 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:31.457964 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 15:01:31.458267 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260809 15:01:31.459856 1484 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45614
W20260809 15:01:31.460846 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1295: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:31.460975 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 1295
I20260809 15:01:31.479755 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1499
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287691, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:31.493923 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:31.480659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.27.186.131:44257)
0809 15:01:31.481243 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:31.481249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:31.481276 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:01:31.481331 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:31.481337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:31.481517 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:31.481630 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:31.482804 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.483808 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.484920 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:31.485441 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:31.485539 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:31.489218 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:31.489229 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:31.489241 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:31.489272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:31.491835 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.492352 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.492358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.492360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.492468 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:31.492704 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:31.492707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:31.492709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:31.493017 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:31.493236 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:31.493398 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 15:01:31.493579 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":283,"threads_started":1}
W20260809 15:01:31.495100 1677 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47358
W20260809 15:01:31.496085 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:01:31.496225 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 1499
I20260809 15:01:31.515355 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 945
I20260809 15:01:31.516350 1034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:31.619235 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 945
2026-08-09T15:01:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3554 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[1709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[1709](info): set up 2 sockets
Aug 09 15:01:31 dist-test-slave-q9h9 krb5kdc[1709](info): commencing operation
krb5kdc: starting...
W20260809 15:01:32.805928 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:32 dist-test-slave-q9h9 krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:32.980324 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:41563
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:41563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:33.285408 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:33.285651 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:33.285775 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:33.292692 1725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:33.292783 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:33.292855 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:33.292925 1725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:33.292981 1725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:33.302218 1725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:41563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:41563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:33.303363 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:33.304582 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:33.315444 1737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:33.315551 1738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:33.315470 1742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:33.316459 1725 server_base.cc:1034] running on GCE node
I20260809 15:01:33.317166 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:33.318459 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:33.319655 1725 hybrid_clock.cc:648] HybridClock initialized: now 1786287693319604 us; error 38 us; skew 500 ppm
I20260809 15:01:33.319959 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:33 dist-test-slave-q9h9 krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:33.323089 1725 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:33.324393 1725 webserver.cc:460] Webserver started at http://127.27.186.190:36975/ using document root <none> and password file <none>
I20260809 15:01:33.324996 1725 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:33.325112 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:33.325408 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:33.328161 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "7cda3590fe864730a56d5d6d5dcd5b52"
format_stamp: "Formatted at 2026-08-09 15:01:33 on dist-test-slave-q9h9"
server_key: "4a3b519944a5069e1197c307cf2965f1"
server_key_iv: "c1ac18bbe2dda3a4837560cced783cb0"
server_key_version: "encryptionkey@0"
I20260809 15:01:33.328899 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "7cda3590fe864730a56d5d6d5dcd5b52"
format_stamp: "Formatted at 2026-08-09 15:01:33 on dist-test-slave-q9h9"
server_key: "4a3b519944a5069e1197c307cf2965f1"
server_key_iv: "c1ac18bbe2dda3a4837560cced783cb0"
server_key_version: "encryptionkey@0"
I20260809 15:01:33.333577 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260809 15:01:33.336781 1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:33.337901 1725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:33.338096 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "7cda3590fe864730a56d5d6d5dcd5b52"
format_stamp: "Formatted at 2026-08-09 15:01:33 on dist-test-slave-q9h9"
server_key: "4a3b519944a5069e1197c307cf2965f1"
server_key_iv: "c1ac18bbe2dda3a4837560cced783cb0"
server_key_version: "encryptionkey@0"
I20260809 15:01:33.338274 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:33.363997 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:33.367631 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:33.367882 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:33.384461 1725 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:41563
I20260809 15:01:33.384505 1829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:41563 every 8 connection(s)
I20260809 15:01:33.385669 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:33.389027 1831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:33.394822 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 1725
I20260809 15:01:33.395025 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:33.395624 28394 external_mini_cluster.cc:1428] Setting key 60117bb36e8f2cb43bbde92de5034fdb
I20260809 15:01:33.399080 1831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52: Bootstrap starting.
I20260809 15:01:33.402415 1831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:33.403417 1831 log.cc:824] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:33.406137 1831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52: No bootstrap required, opened a new log
Aug 09 15:01:33 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:33.413859 1831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41563 } }
I20260809 15:01:33.414214 1831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:33.414323 1831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cda3590fe864730a56d5d6d5dcd5b52, State: Initialized, Role: FOLLOWER
I20260809 15:01:33.414909 1831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41563 } }
I20260809 15:01:33.415144 1831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:33.415278 1831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:33.415455 1831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:33.417198 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:33.397431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.27.186.190:41563)
0809 15:01:33.398153 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:33.398163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:33.398202 (+ 39us) server_negotiation.cc:407] Connection header received
0809 15:01:33.399846 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:33.399868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:33.400202 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:33.400518 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:33.402271 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:33.403550 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:33.404985 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:33.405611 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:33.405740 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:33.410065 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:33.410076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:33.410098 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:33.410136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:33.413051 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:33.413674 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:33.413681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:33.413686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:33.413857 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:33.414292 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:33.414297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:33.414299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:33.414842 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:33.415148 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:33.415584 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 15:01:33.415967 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":322,"threads_started":1}
I20260809 15:01:33.418270 1831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41563 } }
I20260809 15:01:33.418717 1831 leader_election.cc:302] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cda3590fe864730a56d5d6d5dcd5b52; no voters:
I20260809 15:01:33.419328 1831 leader_election.cc:288] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:33.419425 1838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:33.420125 1838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [term 1 LEADER]: Becoming Leader. State: Replica: 7cda3590fe864730a56d5d6d5dcd5b52, State: Running, Role: LEADER
I20260809 15:01:33.420750 1838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41563 } }
I20260809 15:01:33.421547 1831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:33.423708 1842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cda3590fe864730a56d5d6d5dcd5b52. Latest consensus state: current_term: 1 leader_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41563 } } }
I20260809 15:01:33.423756 1840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cda3590fe864730a56d5d6d5dcd5b52" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41563 } } }
I20260809 15:01:33.424113 1842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:33.424117 1840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:33.425586 1847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:33.431123 1847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:33.439954 1847 catalog_manager.cc:1347] Generated new cluster ID: 76c77e6e2eea4174a0462e83bc13633c
I20260809 15:01:33.440054 1847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:33.451506 1847 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:33.452597 1847 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:33.460840 1847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cda3590fe864730a56d5d6d5dcd5b52: Generated new TSK 0
I20260809 15:01:33.461695 1847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:33.504973 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41563
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:33.818519 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:33.818766 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:33.818897 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:33.825824 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:33.825928 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:33.826090 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:33.835130 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:33.836189 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:33.837313 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:33.849447 1874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:33.849458 1876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:33.849907 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:33.850322 1866 server_base.cc:1034] running on GCE node
I20260809 15:01:33.850852 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:33.851635 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:33.852833 1866 hybrid_clock.cc:648] HybridClock initialized: now 1786287693852779 us; error 45 us; skew 500 ppm
I20260809 15:01:33.853097 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:33 dist-test-slave-q9h9 krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:33.855960 1866 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:33.857205 1866 webserver.cc:460] Webserver started at http://127.27.186.129:39103/ using document root <none> and password file <none>
I20260809 15:01:33.857795 1866 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:33.857910 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:33.858242 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:33.860827 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "d4b61da7081f415e8a57232710e3d29c"
format_stamp: "Formatted at 2026-08-09 15:01:33 on dist-test-slave-q9h9"
server_key: "3c6dfdb02447e869ab2020d93577eb45"
server_key_iv: "6a2c03c3badc8c9fa93a37d55ae9e9f9"
server_key_version: "encryptionkey@0"
I20260809 15:01:33.861568 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "d4b61da7081f415e8a57232710e3d29c"
format_stamp: "Formatted at 2026-08-09 15:01:33 on dist-test-slave-q9h9"
server_key: "3c6dfdb02447e869ab2020d93577eb45"
server_key_iv: "6a2c03c3badc8c9fa93a37d55ae9e9f9"
server_key_version: "encryptionkey@0"
I20260809 15:01:33.866179 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:01:33.869261 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:33.870469 1866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:01:33.870617 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "d4b61da7081f415e8a57232710e3d29c"
format_stamp: "Formatted at 2026-08-09 15:01:33 on dist-test-slave-q9h9"
server_key: "3c6dfdb02447e869ab2020d93577eb45"
server_key_iv: "6a2c03c3badc8c9fa93a37d55ae9e9f9"
server_key_version: "encryptionkey@0"
I20260809 15:01:33.870810 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:33.882931 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:33.886253 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:33.886497 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:33.887342 1866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:33.888543 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:33.888620 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:33.888741 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:33.888801 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:33.918356 1866 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:34869
I20260809 15:01:33.918423 2055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:34869 every 8 connection(s)
I20260809 15:01:33.919428 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:33.920956 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 1866
I20260809 15:01:33.921142 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:33.921842 28394 external_mini_cluster.cc:1428] Setting key 1647d79a0e6dc243810a0af31f5dc16f
Aug 09 15:01:33 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:33.942348 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:33.921739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:39823 (local address 127.27.186.190:41563)
0809 15:01:33.922249 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:33.922255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:33.926749 (+ 4494us) server_negotiation.cc:407] Connection header received
0809 15:01:33.928488 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:33.928493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:33.928565 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:33.928667 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:33.930039 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:33.931004 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:33.932549 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:33.932954 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:33.933060 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:33.936250 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:33.936256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:33.936260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:33.936306 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:33.939366 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:33.940015 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:33.940021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:33.940025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:33.940174 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:33.940557 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:33.940562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:33.940565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:33.940910 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:33.941075 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:33.941689 (+ 614us) server_negotiation.cc:299] Negotiation successful
0809 15:01:33.941907 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
I20260809 15:01:33.943044 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:33.922248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41563 (local address 127.27.186.129:39823)
0809 15:01:33.926580 (+ 4332us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:33.926629 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:33.928234 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:33.928850 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:33.928856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:33.929350 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:33.929857 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:33.929873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:33.931151 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:33.931156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:33.932420 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:33.932434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:33.933163 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:33.933171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:33.933392 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:33.933965 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:33.933985 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:33.936027 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:33.939502 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:33.939508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:33.939517 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:33.939902 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:33.940272 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:33.940275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:33.940277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:33.940457 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:33.941032 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:33.941039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:33.941362 (+ 323us) client_negotiation.cc:769] Sending connection context
0809 15:01:33.941565 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 15:01:33.941895 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4072,"thread_start_us":3858,"threads_started":1}
I20260809 15:01:33.944355 2057 heartbeater.cc:347] Connected to a master server at 127.27.186.190:41563
I20260809 15:01:33.944628 2057 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
I20260809 15:01:33.945384 2057 heartbeater.cc:511] Master 127.27.186.190:41563 requested a full tablet report, sending...
I20260809 15:01:33.947320 1781 ts_manager.cc:194] Registered new tserver with Master: d4b61da7081f415e8a57232710e3d29c (127.27.186.129:34869)
I20260809 15:01:33.948675 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:39823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:33.964519 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41563
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:34.275067 2072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:34.275311 2072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:34.275410 2072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:34.282197 2072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:34.282282 2072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:34.282392 2072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:34.291473 2072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:34.292539 2072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:34.293725 2072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:34.305620 2079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:34.305706 2081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:34.305887 2083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:34.306828 2072 server_base.cc:1034] running on GCE node
I20260809 15:01:34.307363 2072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:34.308158 2072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:34.309357 2072 hybrid_clock.cc:648] HybridClock initialized: now 1786287694309308 us; error 42 us; skew 500 ppm
I20260809 15:01:34.309614 2072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:34.312594 2072 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:34.313830 2072 webserver.cc:460] Webserver started at http://127.27.186.130:34727/ using document root <none> and password file <none>
I20260809 15:01:34.314460 2072 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:34.314559 2072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:34.314801 2072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:34.317451 2072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "ec632b822a814a7ab06478867787be6b"
format_stamp: "Formatted at 2026-08-09 15:01:34 on dist-test-slave-q9h9"
server_key: "fe0a49b31cc7184872df95ace4f92554"
server_key_iv: "d70bf426d0d236292c7ede2f86782183"
server_key_version: "encryptionkey@0"
I20260809 15:01:34.318200 2072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "ec632b822a814a7ab06478867787be6b"
format_stamp: "Formatted at 2026-08-09 15:01:34 on dist-test-slave-q9h9"
server_key: "fe0a49b31cc7184872df95ace4f92554"
server_key_iv: "d70bf426d0d236292c7ede2f86782183"
server_key_version: "encryptionkey@0"
I20260809 15:01:34.322925 2072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:01:34.326146 2091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:34.327304 2072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:01:34.327436 2072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "ec632b822a814a7ab06478867787be6b"
format_stamp: "Formatted at 2026-08-09 15:01:34 on dist-test-slave-q9h9"
server_key: "fe0a49b31cc7184872df95ace4f92554"
server_key_iv: "d70bf426d0d236292c7ede2f86782183"
server_key_version: "encryptionkey@0"
I20260809 15:01:34.327612 2072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:34.344173 2072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:34.347734 2072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:34.348050 2072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:34.349117 2072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:34.350740 2072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:34.350818 2072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:34.350932 2072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:34.350984 2072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:34.378368 2072 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:35173
I20260809 15:01:34.378409 2272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:35173 every 8 connection(s)
I20260809 15:01:34.379567 2072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:34.389861 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 2072
I20260809 15:01:34.390074 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:34.390470 28394 external_mini_cluster.cc:1428] Setting key d420639936ed326258f5bf86ced30f7e
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:34.399009 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.381932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:55851 (local address 127.27.186.190:41563)
0809 15:01:34.382165 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:34.382171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:34.383294 (+ 1123us) server_negotiation.cc:407] Connection header received
0809 15:01:34.385211 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:34.385216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:34.385326 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:34.385419 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:34.386898 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.387621 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.389103 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.389640 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.389767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.393486 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:34.393491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:34.393494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:34.393524 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:34.396117 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.396782 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.396787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.396789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.396930 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.397335 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.397338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.397341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.397582 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:34.397738 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:34.398417 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 15:01:34.398666 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 15:01:34.399912 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.382437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41563 (local address 127.27.186.130:55851)
0809 15:01:34.383134 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:34.383176 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:34.384961 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:34.385657 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:34.385664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:34.386212 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:34.386729 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:34.386746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.387789 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.387793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:34.388968 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:34.388980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.389881 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.389888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:34.390191 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.390886 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:34.390919 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:34.393291 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:34.396246 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:34.396255 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:34.396269 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:34.396657 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:34.397033 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:34.397038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:34.397042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:34.397217 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:34.397703 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:34.397711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:34.398061 (+ 350us) client_negotiation.cc:769] Sending connection context
0809 15:01:34.398302 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 15:01:34.398764 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":278,"threads_started":1}
I20260809 15:01:34.401262 2274 heartbeater.cc:347] Connected to a master server at 127.27.186.190:41563
I20260809 15:01:34.401582 2274 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:34.402307 2274 heartbeater.cc:511] Master 127.27.186.190:41563 requested a full tablet report, sending...
I20260809 15:01:34.403577 1781 ts_manager.cc:194] Registered new tserver with Master: ec632b822a814a7ab06478867787be6b (127.27.186.130:35173)
I20260809 15:01:34.404335 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:55851
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:34.433509 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41563
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:34.737844 2283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:34.738108 2283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:34.738214 2283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:34.745043 2283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:34.745133 2283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:34.745239 2283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:34.754172 2283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:34.755185 2283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:34.756304 2283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:34.768258 2289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:34.768303 2293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:34.768213 2290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:34.769306 2283 server_base.cc:1034] running on GCE node
I20260809 15:01:34.769833 2283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:34.770685 2283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:34.771875 2283 hybrid_clock.cc:648] HybridClock initialized: now 1786287694771829 us; error 41 us; skew 500 ppm
I20260809 15:01:34.772145 2283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:34.775195 2283 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:34.776441 2283 webserver.cc:460] Webserver started at http://127.27.186.131:34081/ using document root <none> and password file <none>
I20260809 15:01:34.777034 2283 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:34.777148 2283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:34.777426 2283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:34.780054 2283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "71f19cf0ca594a508d2cca3ad49e666b"
format_stamp: "Formatted at 2026-08-09 15:01:34 on dist-test-slave-q9h9"
server_key: "7ae3aed8f7e188a8d973a52e8051fe5e"
server_key_iv: "c4771a1280c924f8d380cf3fa5a860ba"
server_key_version: "encryptionkey@0"
I20260809 15:01:34.780759 2283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "71f19cf0ca594a508d2cca3ad49e666b"
format_stamp: "Formatted at 2026-08-09 15:01:34 on dist-test-slave-q9h9"
server_key: "7ae3aed8f7e188a8d973a52e8051fe5e"
server_key_iv: "c4771a1280c924f8d380cf3fa5a860ba"
server_key_version: "encryptionkey@0"
I20260809 15:01:34.785429 2283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:34.788594 2303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:34.789749 2283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:34.789903 2283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "71f19cf0ca594a508d2cca3ad49e666b"
format_stamp: "Formatted at 2026-08-09 15:01:34 on dist-test-slave-q9h9"
server_key: "7ae3aed8f7e188a8d973a52e8051fe5e"
server_key_iv: "c4771a1280c924f8d380cf3fa5a860ba"
server_key_version: "encryptionkey@0"
I20260809 15:01:34.790100 2283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:34.823230 2283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:34.827092 2283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:34.827338 2283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:34.828190 2283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:34.829388 2283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:34.829470 2283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:34.829591 2283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:34.829643 2283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:34.866245 2283 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:37377
I20260809 15:01:34.866314 2490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:37377 every 8 connection(s)
I20260809 15:01:34.867472 2283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:34.877388 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 2283
I20260809 15:01:34.877566 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:34.877970 28394 external_mini_cluster.cc:1428] Setting key 50c984f2ddcba282f3598f04aa7bd474
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:34.888831 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.869869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:51545 (local address 127.27.186.190:41563)
0809 15:01:34.870092 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:34.870098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:34.871302 (+ 1204us) server_negotiation.cc:407] Connection header received
0809 15:01:34.873816 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:34.873821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:34.873906 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:34.873995 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:34.875888 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.876582 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.878808 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.879370 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.879501 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.883491 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:34.883496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:34.883499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:34.883530 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:34.885986 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.886817 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.886821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.886823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.886952 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.887383 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.887386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.887387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.887617 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:34.887721 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:34.888443 (+ 722us) server_negotiation.cc:299] Negotiation successful
0809 15:01:34.888590 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:01:34.890069 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.870367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41563 (local address 127.27.186.131:51545)
0809 15:01:34.871120 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:34.871170 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:34.873462 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:34.874274 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:34.874285 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:34.874994 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:34.875722 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:34.875746 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.876740 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.876745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:34.878665 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:34.878680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.879600 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.879608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:34.879889 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.880591 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:34.880625 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:34.883269 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:34.886135 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:34.886143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:34.886157 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:34.886711 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:34.887043 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:34.887048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:34.887051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:34.887292 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:34.887721 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:34.887728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:34.888095 (+ 367us) client_negotiation.cc:769] Sending connection context
0809 15:01:34.888383 (+ 288us) client_negotiation.cc:240] Negotiation successful
0809 15:01:34.888838 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":289,"threads_started":1}
I20260809 15:01:34.891343 2492 heartbeater.cc:347] Connected to a master server at 127.27.186.190:41563
I20260809 15:01:34.891664 2492 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:34.892457 2492 heartbeater.cc:511] Master 127.27.186.190:41563 requested a full tablet report, sending...
I20260809 15:01:34.893909 1781 ts_manager.cc:194] Registered new tserver with Master: 71f19cf0ca594a508d2cca3ad49e666b (127.27.186.131:37377)
I20260809 15:01:34.894773 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:51545
I20260809 15:01:34.904951 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260809 15:01:34.951277 2057 heartbeater.cc:503] Master 127.27.186.190:41563 was elected leader, sending a full tablet report...
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:34.958544 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.944454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.27.186.129:34869)
0809 15:01:34.945023 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:34.945031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:34.945062 (+ 31us) server_negotiation.cc:407] Connection header received
0809 15:01:34.945185 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:34.945192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:34.945370 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:34.945506 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:34.947015 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.948006 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.949190 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.949710 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.949821 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.953653 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:34.953663 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:34.953675 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:34.953708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:34.956386 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.956959 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.956965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.956967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.957084 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.957349 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.957352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.957354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.957650 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:34.957877 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:34.958084 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 15:01:34.958278 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
Aug 09 15:01:34 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:34.974093 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.962204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.27.186.190:41563)
0809 15:01:34.962394 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:34.962399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:34.962623 (+ 224us) server_negotiation.cc:407] Connection header received
0809 15:01:34.962792 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:34.962796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:34.962855 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:34.962922 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:34.964095 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.964765 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.965920 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.966344 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.966433 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.969725 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:34.969730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:34.969733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:34.969761 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:34.972058 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.972536 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.972539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.972542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.972663 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.972920 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.972923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.972925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.973213 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:34.973387 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:34.973605 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 15:01:34.973746 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 15:01:34.974555 1781 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46074
I20260809 15:01:34.975432 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1866
I20260809 15:01:34.987495 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:34.976483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.27.186.129:34869)
0809 15:01:34.976665 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:34.976671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:34.976765 (+ 94us) server_negotiation.cc:407] Connection header received
0809 15:01:34.976921 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:34.976926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:34.976987 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:34.977061 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:34.978191 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.978838 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.980006 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:34.980415 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:34.980516 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:34.982881 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:34.982885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:34.982888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:34.982916 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:34.985191 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.985768 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.985773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.985777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.985893 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:34.986252 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:34.986257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:34.986261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:34.986493 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:34.986718 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:34.987001 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 15:01:34.987208 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:01:34.988008 2050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:35.072692 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 1866
I20260809 15:01:35.093446 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 2072
Aug 09 15:01:35 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:35.108352 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:35.094322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.27.186.130:35173)
0809 15:01:35.094853 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:35.094863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:35.094890 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:01:35.094959 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:35.094966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:35.095174 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:35.095314 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:35.096391 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:35.097360 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:35.098568 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:35.099094 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:35.099190 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:35.102890 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:35.102903 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:35.102917 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:35.102950 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:35.105835 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:35.106541 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:35.106549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:35.106551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:35.106705 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:35.107063 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:35.107066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:35.107067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:35.107391 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:35.107609 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:35.107871 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 15:01:35.108056 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":260,"threads_started":1}
I20260809 15:01:35.109579 2268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:35.189623 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 2072
I20260809 15:01:35.217339 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 2283
Aug 09 15:01:35 dist-test-slave-q9h9 krb5kdc[1709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:35.233208 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:35.218277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.27.186.131:37377)
0809 15:01:35.218932 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:35.218939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:35.218969 (+ 30us) server_negotiation.cc:407] Connection header received
0809 15:01:35.219032 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:35.219038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:35.219241 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:35.219397 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:35.220683 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:35.221727 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:35.223029 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:35.223573 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:35.223698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:35.227670 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:35.227682 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:35.227696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:35.227732 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:35.230549 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:35.231308 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:35.231315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:35.231340 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:35.231482 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:35.231817 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:35.231820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:35.231822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:35.232179 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:35.232447 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:35.232713 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 15:01:35.232920 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260809 15:01:35.234710 2482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:35.315614 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 2283
I20260809 15:01:35.335659 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 1725
I20260809 15:01:35.336750 1819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:35.437208 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 1725
2026-08-09T15:01:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3816 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:35 dist-test-slave-q9h9 krb5kdc[2522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:35 dist-test-slave-q9h9 krb5kdc[2522](info): set up 2 sockets
Aug 09 15:01:35 dist-test-slave-q9h9 krb5kdc[2522](info): commencing operation
krb5kdc: starting...
W20260809 15:01:36.624099 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:36 dist-test-slave-q9h9 krb5kdc[2522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:36.798193 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43483
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:37.098474 2547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:37.098719 2547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:37.098812 2547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:37.105597 2547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:37.105685 2547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:37.105736 2547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:37.105790 2547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:37.105836 2547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:37.114884 2547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:43483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:37.115891 2547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:37.117003 2547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:37.127358 2557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:37.127434 2558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:37.127580 2561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:37.128293 2547 server_base.cc:1034] running on GCE node
I20260809 15:01:37.128971 2547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:37.130232 2547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:37.131438 2547 hybrid_clock.cc:648] HybridClock initialized: now 1786287697131388 us; error 40 us; skew 500 ppm
I20260809 15:01:37.131757 2547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:37 dist-test-slave-q9h9 krb5kdc[2522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287697, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:37.134677 2547 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:37.135686 2547 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:37.135774 2547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:37.136025 2547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:37.138963 2547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "0bcbf15aa60042679f84e889abdf8855"
format_stamp: "Formatted at 2026-08-09 15:01:37 on dist-test-slave-q9h9"
server_key: "e1051366a287a2027d0bc9dd8a5be015"
server_key_iv: "aa76320bf523aa94969fac46f0d51767"
server_key_version: "encryptionkey@0"
I20260809 15:01:37.139663 2547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "0bcbf15aa60042679f84e889abdf8855"
format_stamp: "Formatted at 2026-08-09 15:01:37 on dist-test-slave-q9h9"
server_key: "e1051366a287a2027d0bc9dd8a5be015"
server_key_iv: "aa76320bf523aa94969fac46f0d51767"
server_key_version: "encryptionkey@0"
I20260809 15:01:37.144289 2547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:37.147430 2572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:37.148622 2547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:37.148761 2547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "0bcbf15aa60042679f84e889abdf8855"
format_stamp: "Formatted at 2026-08-09 15:01:37 on dist-test-slave-q9h9"
server_key: "e1051366a287a2027d0bc9dd8a5be015"
server_key_iv: "aa76320bf523aa94969fac46f0d51767"
server_key_version: "encryptionkey@0"
I20260809 15:01:37.148942 2547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:37.165652 2547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:37.169394 2547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:37.169591 2547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:37.186077 2547 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:43483
I20260809 15:01:37.186084 2643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:43483 every 8 connection(s)
I20260809 15:01:37.186847 2547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:37.190210 2645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:37.191109 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 2547
I20260809 15:01:37.191354 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:37.192159 28394 external_mini_cluster.cc:1428] Setting key cb2f394c88ad88285721e3f7a071ca3f
I20260809 15:01:37.203768 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855: Bootstrap starting.
Aug 09 15:01:37 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:37.207343 2645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:37.208348 2645 log.cc:824] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:37.211261 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855: No bootstrap required, opened a new log
I20260809 15:01:37.214787 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:37.194086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.27.186.190:43483)
0809 15:01:37.194839 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:37.194850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:37.194891 (+ 41us) server_negotiation.cc:407] Connection header received
0809 15:01:37.196466 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:37.196486 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:37.196845 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:37.197148 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:37.199074 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:37.200310 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:37.202059 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:37.202675 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:37.202808 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:37.207326 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:37.207336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:37.207348 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:37.207380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:37.210642 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:37.211271 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:37.211279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:37.211287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:37.211474 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:37.211824 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:37.211829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:37.211832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:37.212531 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:37.212800 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:37.213129 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 15:01:37.213540 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":331,"threads_started":1}
I20260809 15:01:37.220196 2645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcbf15aa60042679f84e889abdf8855" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43483 } }
I20260809 15:01:37.220504 2645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:37.220602 2645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcbf15aa60042679f84e889abdf8855, State: Initialized, Role: FOLLOWER
I20260809 15:01:37.221100 2645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcbf15aa60042679f84e889abdf8855" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43483 } }
I20260809 15:01:37.221307 2645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:37.221428 2645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:37.221575 2645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:37.223757 2645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcbf15aa60042679f84e889abdf8855" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43483 } }
I20260809 15:01:37.224238 2645 leader_election.cc:302] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bcbf15aa60042679f84e889abdf8855; no voters:
I20260809 15:01:37.224727 2645 leader_election.cc:288] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:37.224830 2656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:37.225736 2656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [term 1 LEADER]: Becoming Leader. State: Replica: 0bcbf15aa60042679f84e889abdf8855, State: Running, Role: LEADER
I20260809 15:01:37.226223 2656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcbf15aa60042679f84e889abdf8855" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43483 } }
I20260809 15:01:37.226460 2645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:37.228943 2657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bcbf15aa60042679f84e889abdf8855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcbf15aa60042679f84e889abdf8855" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43483 } } }
I20260809 15:01:37.229017 2658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bcbf15aa60042679f84e889abdf8855. Latest consensus state: current_term: 1 leader_uuid: "0bcbf15aa60042679f84e889abdf8855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcbf15aa60042679f84e889abdf8855" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43483 } } }
I20260809 15:01:37.229319 2658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:37.229328 2657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:37.233675 2666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:37.237716 2666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:37.245038 2666 catalog_manager.cc:1347] Generated new cluster ID: 2624b16220ea47a2a010e18a664c71f0
I20260809 15:01:37.245134 2666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:37.287437 2666 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:37.288832 2666 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:37.300730 2666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855: Generated new TSK 0
I20260809 15:01:37.301431 2666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:37.347683 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43483
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:37.664726 2690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:37.664959 2690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:37.665081 2690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:37.672017 2690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:37.672109 2690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:37.672259 2690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:37.681469 2690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:37.682663 2690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:37.683900 2690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:37.696089 2698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:37.696130 2702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:37.696099 2700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:37.696952 2690 server_base.cc:1034] running on GCE node
I20260809 15:01:37.697468 2690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:37.698354 2690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:37.699546 2690 hybrid_clock.cc:648] HybridClock initialized: now 1786287697699501 us; error 41 us; skew 500 ppm
I20260809 15:01:37.699836 2690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:37 dist-test-slave-q9h9 krb5kdc[2522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287697, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:37.703022 2690 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:37.704033 2690 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:37.704118 2690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:37.704376 2690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:37.707116 2690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "a1d7d822680c4628b1dc6199ba9b511b"
format_stamp: "Formatted at 2026-08-09 15:01:37 on dist-test-slave-q9h9"
server_key: "69427bd4d02adc18de77e862a07c34b9"
server_key_iv: "6cff36c127f89405e98bd716b90e6998"
server_key_version: "encryptionkey@0"
I20260809 15:01:37.707880 2690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "a1d7d822680c4628b1dc6199ba9b511b"
format_stamp: "Formatted at 2026-08-09 15:01:37 on dist-test-slave-q9h9"
server_key: "69427bd4d02adc18de77e862a07c34b9"
server_key_iv: "6cff36c127f89405e98bd716b90e6998"
server_key_version: "encryptionkey@0"
I20260809 15:01:37.712636 2690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:37.715938 2712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:37.717059 2690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:37.717219 2690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "a1d7d822680c4628b1dc6199ba9b511b"
format_stamp: "Formatted at 2026-08-09 15:01:37 on dist-test-slave-q9h9"
server_key: "69427bd4d02adc18de77e862a07c34b9"
server_key_iv: "6cff36c127f89405e98bd716b90e6998"
server_key_version: "encryptionkey@0"
I20260809 15:01:37.717403 2690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:37.733480 2690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:37.736917 2690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:37.737144 2690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:37.737989 2690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:37.739259 2690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:37.739346 2690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:37.739451 2690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:37.739528 2690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:37.768764 2690 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:45905
I20260809 15:01:37.768821 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:45905 every 8 connection(s)
I20260809 15:01:37.769500 2690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:37.774974 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 2690
I20260809 15:01:37.775194 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:37.775832 28394 external_mini_cluster.cc:1428] Setting key 436851fefa00f632f45dc2488a561e93
Aug 09 15:01:37 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287697, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:37.788580 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:37.771692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:40165 (local address 127.27.186.190:43483)
0809 15:01:37.772126 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:37.772131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:37.772987 (+ 856us) server_negotiation.cc:407] Connection header received
0809 15:01:37.774689 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:37.774696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:37.774784 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:37.774920 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:37.776381 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:37.777447 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:37.778939 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:37.779526 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:37.779644 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:37.783033 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:37.783037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:37.783041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:37.783073 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:37.785729 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:37.786473 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:37.786479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:37.786482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:37.786627 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:37.786990 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:37.786995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:37.786999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:37.787319 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:37.787459 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:37.788019 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 15:01:37.788233 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":228,"threads_started":1}
I20260809 15:01:37.789148 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:37.772176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43483 (local address 127.27.186.129:40165)
0809 15:01:37.772829 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:37.772870 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:37.774438 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:37.775074 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:37.775084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:37.775637 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:37.776178 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:37.776209 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:37.777587 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:37.777591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:37.778802 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:37.778815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:37.779730 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:37.779734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:37.779907 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:37.780488 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:37.780508 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:37.782816 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:37.785882 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:37.785889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:37.785899 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:37.786349 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:37.786726 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:37.786729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:37.786731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:37.786892 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:37.787417 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:37.787422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:37.787682 (+ 260us) client_negotiation.cc:769] Sending connection context
0809 15:01:37.787882 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 15:01:37.788210 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":259,"threads_started":1}
I20260809 15:01:37.790624 2893 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43483
I20260809 15:01:37.790896 2893 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:37.791487 2893 heartbeater.cc:511] Master 127.27.186.190:43483 requested a full tablet report, sending...
I20260809 15:01:37.793426 2595 ts_manager.cc:194] Registered new tserver with Master: a1d7d822680c4628b1dc6199ba9b511b (127.27.186.129:45905)
I20260809 15:01:37.794847 2595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:40165
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:37.818538 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43483
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:38.125759 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:38.125999 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:38.126153 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:38.133085 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:38.133183 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:38.133328 2902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:38.142374 2902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:38.143474 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:38.144644 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:38.156620 2908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:38.156672 2909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:38.156695 2914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:38.157533 2902 server_base.cc:1034] running on GCE node
I20260809 15:01:38.158167 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:38.159049 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:38.160249 2902 hybrid_clock.cc:648] HybridClock initialized: now 1786287698160184 us; error 50 us; skew 500 ppm
I20260809 15:01:38.160543 2902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:38 dist-test-slave-q9h9 krb5kdc[2522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:38.166872 2902 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:38.167901 2902 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:38.168002 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:38.168294 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:38.171064 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "9d8564c664b04de3ac0999b4b2c1fb31"
format_stamp: "Formatted at 2026-08-09 15:01:38 on dist-test-slave-q9h9"
server_key: "128ac418b7a7259096220aefa75fea48"
server_key_iv: "337a4a92188bbf6c080ff9e5444dc7e5"
server_key_version: "encryptionkey@0"
I20260809 15:01:38.171844 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "9d8564c664b04de3ac0999b4b2c1fb31"
format_stamp: "Formatted at 2026-08-09 15:01:38 on dist-test-slave-q9h9"
server_key: "128ac418b7a7259096220aefa75fea48"
server_key_iv: "337a4a92188bbf6c080ff9e5444dc7e5"
server_key_version: "encryptionkey@0"
I20260809 15:01:38.176546 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 15:01:38.179821 2922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:38.180971 2902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:38.181130 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "9d8564c664b04de3ac0999b4b2c1fb31"
format_stamp: "Formatted at 2026-08-09 15:01:38 on dist-test-slave-q9h9"
server_key: "128ac418b7a7259096220aefa75fea48"
server_key_iv: "337a4a92188bbf6c080ff9e5444dc7e5"
server_key_version: "encryptionkey@0"
I20260809 15:01:38.181318 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:38.202571 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:38.207041 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:38.207342 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:38.208153 2902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:38.209331 2902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:38.209427 2902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:38.209501 2902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:38.209558 2902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:38.239439 2902 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:34249
I20260809 15:01:38.239491 3105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:34249 every 8 connection(s)
I20260809 15:01:38.240222 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:38.242522 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 2902
I20260809 15:01:38.242733 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:38.243255 28394 external_mini_cluster.cc:1428] Setting key 38a0ee329d8d0fbabc0820c58d75c062
Aug 09 15:01:38 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:38.259582 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.242529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:44825 (local address 127.27.186.190:43483)
0809 15:01:38.242691 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.242695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.244090 (+ 1395us) server_negotiation.cc:407] Connection header received
0809 15:01:38.246195 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.246200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.246274 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.246368 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:38.247737 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.248686 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.250171 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.250706 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.250823 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.254039 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:38.254047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:38.254052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:38.254091 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:38.256565 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.257413 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.257418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.257420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.257552 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.257996 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.257998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.258000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.258296 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:38.258410 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.259196 (+ 786us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.259356 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:01:38.260841 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.243132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43483 (local address 127.27.186.130:44825)
0809 15:01:38.243927 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.243973 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.245923 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.246533 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.246540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.247050 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:38.247570 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.247592 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.248808 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.248813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.249997 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.250009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.250886 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.250891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.251071 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.251680 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:38.251709 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:38.253800 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:38.256719 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:38.256727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:38.256740 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:38.257303 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:38.257670 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:38.257673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:38.257677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:38.257904 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:38.258421 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:38.258428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:38.258847 (+ 419us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.259128 (+ 281us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.259592 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":329,"threads_started":1}
I20260809 15:01:38.262217 3106 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43483
I20260809 15:01:38.262516 3106 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:38.263207 3106 heartbeater.cc:511] Master 127.27.186.190:43483 requested a full tablet report, sending...
I20260809 15:01:38.264556 2595 ts_manager.cc:194] Registered new tserver with Master: 9d8564c664b04de3ac0999b4b2c1fb31 (127.27.186.130:34249)
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
I20260809 15:01:38.265360 2595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:44825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:38.289077 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43483
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:38.596316 3116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:38.596551 3116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:38.596673 3116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:38.603446 3116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:38.603549 3116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:38.603695 3116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:38.612612 3116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:38.613626 3116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:38.614811 3116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:38.627000 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:38.627106 3129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:38.627036 3127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:38.628371 3116 server_base.cc:1034] running on GCE node
I20260809 15:01:38.628942 3116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:38.629803 3116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:38.631006 3116 hybrid_clock.cc:648] HybridClock initialized: now 1786287698630937 us; error 56 us; skew 500 ppm
I20260809 15:01:38.631300 3116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:38 dist-test-slave-q9h9 krb5kdc[2522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:38.634362 3116 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:38.635354 3116 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:38.635442 3116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:38.635687 3116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:38.638303 3116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "d6e548d592744268b2159fb1a7280694"
format_stamp: "Formatted at 2026-08-09 15:01:38 on dist-test-slave-q9h9"
server_key: "d2a791284e52b6c5164256f435e322ee"
server_key_iv: "d3bde6aa18e3f6652ed78cd979dd3a5a"
server_key_version: "encryptionkey@0"
I20260809 15:01:38.639009 3116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "d6e548d592744268b2159fb1a7280694"
format_stamp: "Formatted at 2026-08-09 15:01:38 on dist-test-slave-q9h9"
server_key: "d2a791284e52b6c5164256f435e322ee"
server_key_iv: "d3bde6aa18e3f6652ed78cd979dd3a5a"
server_key_version: "encryptionkey@0"
I20260809 15:01:38.643580 3116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:38.646763 3139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:38.647938 3116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:38.648084 3116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "d6e548d592744268b2159fb1a7280694"
format_stamp: "Formatted at 2026-08-09 15:01:38 on dist-test-slave-q9h9"
server_key: "d2a791284e52b6c5164256f435e322ee"
server_key_iv: "d3bde6aa18e3f6652ed78cd979dd3a5a"
server_key_version: "encryptionkey@0"
I20260809 15:01:38.648283 3116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:38.668315 3116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:38.671986 3116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:38.672219 3116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:38.673077 3116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:38.674423 3116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:38.674518 3116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:38.674621 3116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:38.674691 3116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:38.704864 3116 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:36611
I20260809 15:01:38.704942 3326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:36611 every 8 connection(s)
I20260809 15:01:38.705612 3116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:38.714542 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 3116
I20260809 15:01:38.714780 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:38.715284 28394 external_mini_cluster.cc:1428] Setting key f88dbb0264789cef3c687cde1fc908c4
Aug 09 15:01:38 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:38.724416 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.707813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:51697 (local address 127.27.186.190:43483)
0809 15:01:38.707979 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.707983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.709014 (+ 1031us) server_negotiation.cc:407] Connection header received
0809 15:01:38.710860 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.710864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.710935 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.711019 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:38.712409 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.713124 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.714972 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.715413 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.715507 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.719431 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:38.719435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:38.719438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:38.719471 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:38.721832 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.722546 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.722550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.722552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.722663 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.723090 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.723094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.723096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.723354 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:38.723462 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.724020 (+ 558us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.724168 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:01:38.725404 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.708165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43483 (local address 127.27.186.131:51697)
0809 15:01:38.708843 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.708888 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.710615 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.711225 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.711232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.711726 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:38.712274 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.712292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.713246 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.713250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.714814 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.714846 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.715651 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.715660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.715923 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.716637 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:38.716678 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:38.719242 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:38.721977 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:38.721984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:38.721994 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:38.722450 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:38.722742 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:38.722745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:38.722747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:38.722917 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:38.723451 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:38.723456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:38.723728 (+ 272us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.723966 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.724307 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":270,"threads_started":1}
I20260809 15:01:38.726675 3327 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43483
I20260809 15:01:38.726933 3327 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:38.727521 3327 heartbeater.cc:511] Master 127.27.186.190:43483 requested a full tablet report, sending...
I20260809 15:01:38.728907 2595 ts_manager.cc:194] Registered new tserver with Master: d6e548d592744268b2159fb1a7280694 (127.27.186.131:36611)
I20260809 15:01:38.730065 2595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:51697
I20260809 15:01:38.730533 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:01:38.746954 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.735553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.27.186.190:43483)
0809 15:01:38.735739 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.735745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.736175 (+ 430us) server_negotiation.cc:407] Connection header received
0809 15:01:38.736305 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.736308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.736360 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.736430 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:38.737827 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.738588 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.739862 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.740238 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.740327 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.742519 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:38.742523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:38.742526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:38.742553 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:38.744797 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.745374 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.745377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.745380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.745501 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.745772 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.745776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.745779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.746096 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:38.746265 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.746544 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.746684 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:01:38.750294 2595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:38.757107 2595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:38.790704 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.767986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.27.186.131:36611)
0809 15:01:38.780448 (+ 12462us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.780457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.781453 (+ 996us) server_negotiation.cc:407] Connection header received
0809 15:01:38.781788 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.781795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.782035 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.782234 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:38.783146 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.784932 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.787921 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.789399 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.789570 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.789730 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.790222 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.790377 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12280,"thread_start_us":12475,"threads_started":1}
I20260809 15:01:38.790712 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.767752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:36611 (local address 127.0.0.1:49070)
0809 15:01:38.781315 (+ 13563us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.781343 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.781512 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.782417 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.782424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.782453 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:38.782882 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.782895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.785109 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.785115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.787719 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.787736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.789592 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.789598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.789893 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.789946 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.790097 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.790284 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13377,"mutex_wait_us":66,"thread_start_us":13270,"threads_started":1}
I20260809 15:01:38.792423 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.768079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45905 (local address 127.0.0.1:56180)
0809 15:01:38.781895 (+ 13816us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.781911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.782038 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.783024 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.783029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.783056 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:38.783471 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.783482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.786605 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.786610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.789889 (+ 3279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.789904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.791605 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.791612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.791775 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.791808 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.791885 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.791999 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13704,"mutex_wait_us":10,"thread_start_us":13546,"threads_started":1}
I20260809 15:01:38.793118 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.768576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.27.186.129:45905)
0809 15:01:38.780805 (+ 12229us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.780811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.782157 (+ 1346us) server_negotiation.cc:407] Connection header received
0809 15:01:38.782364 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.782372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.782573 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.782773 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:38.783662 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.786431 (+ 2769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.790090 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.791419 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.791595 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.791760 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.792412 (+ 652us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.792568 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12104,"thread_start_us":12215,"threads_started":1}
I20260809 15:01:38.794698 3218 tablet_service.cc:1467] Processing CreateTablet for tablet c7367180c5f7482b89daf40e15a7cc70 (DEFAULT_TABLE table=test-table [id=abc6b390dc4a430c8d9b658ff5f51715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:38.796151 3218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7367180c5f7482b89daf40e15a7cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:38.796639 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.782618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34249 (local address 127.0.0.1:58766)
0809 15:01:38.788753 (+ 6135us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.788769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.788873 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.789912 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.789917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.789941 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:38.790383 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.790397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.792134 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.792917 (+ 783us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.794584 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.794597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.795874 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.795881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.796036 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.796066 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.796135 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.796348 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6009,"thread_start_us":6322,"threads_started":1}
I20260809 15:01:38.796675 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.782313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.27.186.130:34249)
0809 15:01:38.789410 (+ 7097us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.789418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.789443 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:38.789532 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.789538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.789727 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.789917 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:38.790586 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.791971 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.794752 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.795688 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.795830 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.795984 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.796227 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.796378 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6952,"thread_start_us":6836,"threads_started":1}
I20260809 15:01:38.796559 2784 tablet_service.cc:1467] Processing CreateTablet for tablet c7367180c5f7482b89daf40e15a7cc70 (DEFAULT_TABLE table=test-table [id=abc6b390dc4a430c8d9b658ff5f51715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:38.797398 2893 heartbeater.cc:503] Master 127.27.186.190:43483 was elected leader, sending a full tablet report...
I20260809 15:01:38.797933 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7367180c5f7482b89daf40e15a7cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:38.800038 3001 tablet_service.cc:1467] Processing CreateTablet for tablet c7367180c5f7482b89daf40e15a7cc70 (DEFAULT_TABLE table=test-table [id=abc6b390dc4a430c8d9b658ff5f51715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:38.800987 3001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7367180c5f7482b89daf40e15a7cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:38.811709 3348 tablet_bootstrap.cc:492] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Bootstrap starting.
I20260809 15:01:38.813577 3347 tablet_bootstrap.cc:492] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Bootstrap starting.
I20260809 15:01:38.815155 3348 tablet_bootstrap.cc:654] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:38.815909 3350 tablet_bootstrap.cc:492] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Bootstrap starting.
I20260809 15:01:38.816414 3348 log.cc:824] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:38.816821 3347 tablet_bootstrap.cc:654] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:38.817950 3347 log.cc:824] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:38.819468 3350 tablet_bootstrap.cc:654] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:38.820338 3348 tablet_bootstrap.cc:492] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: No bootstrap required, opened a new log
I20260809 15:01:38.820348 3347 tablet_bootstrap.cc:492] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: No bootstrap required, opened a new log
I20260809 15:01:38.820605 3347 ts_tablet_manager.cc:1397] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260809 15:01:38.820721 3350 log.cc:824] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:38.821311 3348 ts_tablet_manager.cc:1397] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 15:01:38.823607 3350 tablet_bootstrap.cc:492] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: No bootstrap required, opened a new log
I20260809 15:01:38.823889 3350 ts_tablet_manager.cc:1397] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:01:38.827401 3347 raft_consensus.cc:348] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.827718 3347 raft_consensus.cc:374] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:38.827812 3347 raft_consensus.cc:729] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8564c664b04de3ac0999b4b2c1fb31, State: Initialized, Role: FOLLOWER
I20260809 15:01:38.828336 3347 consensus_queue.cc:260] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.829396 3106 heartbeater.cc:503] Master 127.27.186.190:43483 was elected leader, sending a full tablet report...
I20260809 15:01:38.829952 3347 ts_tablet_manager.cc:1428] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20260809 15:01:38.832353 3348 raft_consensus.cc:348] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.832736 3348 raft_consensus.cc:374] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:38.832841 3348 raft_consensus.cc:729] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e548d592744268b2159fb1a7280694, State: Initialized, Role: FOLLOWER
I20260809 15:01:38.832816 3350 raft_consensus.cc:348] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.833165 3350 raft_consensus.cc:374] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:38.833257 3350 raft_consensus.cc:729] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d7d822680c4628b1dc6199ba9b511b, State: Initialized, Role: FOLLOWER
I20260809 15:01:38.833420 3348 consensus_queue.cc:260] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.833895 3350 consensus_queue.cc:260] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.834637 3327 heartbeater.cc:503] Master 127.27.186.190:43483 was elected leader, sending a full tablet report...
I20260809 15:01:38.835071 3348 ts_tablet_manager.cc:1428] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260809 15:01:38.836390 3350 ts_tablet_manager.cc:1428] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260809 15:01:38.849648 3356 raft_consensus.cc:482] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:38.849844 3356 raft_consensus.cc:504] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.851184 3356 leader_election.cc:288] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1d7d822680c4628b1dc6199ba9b511b (127.27.186.129:45905), 9d8564c664b04de3ac0999b4b2c1fb31 (127.27.186.130:34249)
I20260809 15:01:38.856741 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.851672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:34511 (local address 127.27.186.129:45905)
0809 15:01:38.851854 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.851860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.851881 (+ 21us) server_negotiation.cc:407] Connection header received
0809 15:01:38.851997 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.852002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.852075 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.852164 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:38.852613 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.853594 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.855212 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.855949 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.856084 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.856156 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.856400 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.856516 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 15:01:38.856768 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.851500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45905 (local address 127.27.186.131:34511)
0809 15:01:38.851717 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.851737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.851871 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.852239 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.852243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.852275 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:38.852487 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.852498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.853696 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.853700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.855057 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.855072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.856084 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.856090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.856266 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.856289 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.856367 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.856488 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":20}
I20260809 15:01:38.857594 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.851977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:59253 (local address 127.27.186.130:34249)
0809 15:01:38.852141 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.852147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.852540 (+ 393us) server_negotiation.cc:407] Connection header received
0809 15:01:38.852694 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.852699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.852766 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.852857 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:38.853336 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.854175 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.855817 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.856546 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.856675 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.856740 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.857248 (+ 508us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.857372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 15:01:38.857679 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.851859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34249 (local address 127.27.186.131:59253)
0809 15:01:38.852446 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:38.852462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:38.852576 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:38.852879 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:38.852883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:38.852902 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:38.853198 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.853207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.854309 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.854314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.855671 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:38.855684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.856733 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.856739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:38.856907 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.856939 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:01:38.857210 (+ 271us) client_negotiation.cc:240] Negotiation successful
0809 15:01:38.857330 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":1,"thread_start_us":301,"threads_started":1}
I20260809 15:01:38.857638 2823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7367180c5f7482b89daf40e15a7cc70" candidate_uuid: "d6e548d592744268b2159fb1a7280694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d7d822680c4628b1dc6199ba9b511b" is_pre_election: true
I20260809 15:01:38.858140 2823 raft_consensus.cc:2454] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e548d592744268b2159fb1a7280694 in term 0.
I20260809 15:01:38.858582 3035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7367180c5f7482b89daf40e15a7cc70" candidate_uuid: "d6e548d592744268b2159fb1a7280694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" is_pre_election: true
I20260809 15:01:38.858736 3140 leader_election.cc:302] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d7d822680c4628b1dc6199ba9b511b, d6e548d592744268b2159fb1a7280694; no voters:
I20260809 15:01:38.858969 3035 raft_consensus.cc:2454] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e548d592744268b2159fb1a7280694 in term 0.
I20260809 15:01:38.859078 3356 raft_consensus.cc:2781] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:38.859202 3356 raft_consensus.cc:482] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:38.859304 3356 raft_consensus.cc:3037] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:38.861286 3356 raft_consensus.cc:504] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.861792 3356 leader_election.cc:288] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [CANDIDATE]: Term 1 election: Requested vote from peers a1d7d822680c4628b1dc6199ba9b511b (127.27.186.129:45905), 9d8564c664b04de3ac0999b4b2c1fb31 (127.27.186.130:34249)
I20260809 15:01:38.862167 2823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7367180c5f7482b89daf40e15a7cc70" candidate_uuid: "d6e548d592744268b2159fb1a7280694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d7d822680c4628b1dc6199ba9b511b"
I20260809 15:01:38.862258 3035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7367180c5f7482b89daf40e15a7cc70" candidate_uuid: "d6e548d592744268b2159fb1a7280694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8564c664b04de3ac0999b4b2c1fb31"
I20260809 15:01:38.862323 2823 raft_consensus.cc:3037] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:38.862437 3035 raft_consensus.cc:3037] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:38.864315 2823 raft_consensus.cc:2454] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e548d592744268b2159fb1a7280694 in term 1.
I20260809 15:01:38.864645 3140 leader_election.cc:302] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d7d822680c4628b1dc6199ba9b511b, d6e548d592744268b2159fb1a7280694; no voters:
I20260809 15:01:38.864712 3035 raft_consensus.cc:2454] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e548d592744268b2159fb1a7280694 in term 1.
I20260809 15:01:38.864902 3356 raft_consensus.cc:2781] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:38.865321 3356 raft_consensus.cc:686] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 1 LEADER]: Becoming Leader. State: Replica: d6e548d592744268b2159fb1a7280694, State: Running, Role: LEADER
I20260809 15:01:38.865715 3356 consensus_queue.cc:237] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } }
I20260809 15:01:38.869632 2595 catalog_manager.cc:5495] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e548d592744268b2159fb1a7280694 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "d6e548d592744268b2159fb1a7280694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e548d592744268b2159fb1a7280694" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 36611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 } health_report { overall_health: UNKNOWN } } }
I20260809 15:01:38.887288 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.882510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.27.186.131:36611)
0809 15:01:38.882653 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.882657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.882832 (+ 175us) server_negotiation.cc:407] Connection header received
0809 15:01:38.882986 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.882989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.883051 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.883161 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:38.883533 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.884225 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.885541 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.885964 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.886083 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.886493 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:38.886903 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:38.887000 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.887067 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.887123 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 15:01:38.894264 2823 raft_consensus.cc:1261] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 1 FOLLOWER]: Refusing update from remote peer d6e548d592744268b2159fb1a7280694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:38.894268 3035 raft_consensus.cc:1261] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 1 FOLLOWER]: Refusing update from remote peer d6e548d592744268b2159fb1a7280694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:38.895140 3361 consensus_queue.cc:1035] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8564c664b04de3ac0999b4b2c1fb31" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:38.895435 3356 consensus_queue.cc:1035] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d7d822680c4628b1dc6199ba9b511b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:38.906522 3370 mvcc.cc:205] Tried to move back new op lower bound from 7316634414661976064 to 7316634414557216768. Current Snapshot: MvccSnapshot[applied={T|T < 7316634414661976064}]
I20260809 15:01:38.908483 3369 mvcc.cc:205] Tried to move back new op lower bound from 7316634414661976064 to 7316634414557216768. Current Snapshot: MvccSnapshot[applied={T|T < 7316634414661976064}]
I20260809 15:01:38.918207 2594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:38.918453 2594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:38.921689 2594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bcbf15aa60042679f84e889abdf8855: Sending DeleteTablet for 3 replicas of tablet c7367180c5f7482b89daf40e15a7cc70
I20260809 15:01:38.922724 3218 tablet_service.cc:1514] Processing DeleteTablet for tablet c7367180c5f7482b89daf40e15a7cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:38 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:49070
I20260809 15:01:38.922850 3001 tablet_service.cc:1514] Processing DeleteTablet for tablet c7367180c5f7482b89daf40e15a7cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:38 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:58766
I20260809 15:01:38.923588 3380 tablet_replica.cc:330] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: stopping tablet replica
I20260809 15:01:38.923674 2784 tablet_service.cc:1514] Processing DeleteTablet for tablet c7367180c5f7482b89daf40e15a7cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:38 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:56180
I20260809 15:01:38.923969 3380 raft_consensus.cc:2229] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:38.924559 3380 raft_consensus.cc:2258] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:38.924971 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 2690
I20260809 15:01:38.926657 3380 ts_tablet_manager.cc:1905] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:38.937533 3380 ts_tablet_manager.cc:1918] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:38.937717 3380 log.cc:1196] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/c7367180c5f7482b89daf40e15a7cc70
I20260809 15:01:38.938339 3380 ts_tablet_manager.cc:1939] T c7367180c5f7482b89daf40e15a7cc70 P d6e548d592744268b2159fb1a7280694: Deleting consensus metadata
I20260809 15:01:38.939965 2575 catalog_manager.cc:4841] TS d6e548d592744268b2159fb1a7280694 (127.27.186.131:36611): tablet c7367180c5f7482b89daf40e15a7cc70 (table test-table [id=abc6b390dc4a430c8d9b658ff5f51715]) successfully deleted
I20260809 15:01:38.940313 3381 tablet_replica.cc:330] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: stopping tablet replica
I20260809 15:01:38.940745 3381 raft_consensus.cc:2229] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:38.941262 3381 raft_consensus.cc:2258] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:38.943354 3381 ts_tablet_manager.cc:1905] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:38.944667 3382 tablet_replica.cc:330] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: stopping tablet replica
I20260809 15:01:38.945019 3382 raft_consensus.cc:2229] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:38.945449 3382 raft_consensus.cc:2258] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 09 15:01:38 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:38.949669 3382 ts_tablet_manager.cc:1905] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:38.949867 3381 ts_tablet_manager.cc:1918] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:38.950062 3381 log.cc:1196] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/c7367180c5f7482b89daf40e15a7cc70
I20260809 15:01:38.950582 3381 ts_tablet_manager.cc:1939] T c7367180c5f7482b89daf40e15a7cc70 P 9d8564c664b04de3ac0999b4b2c1fb31: Deleting consensus metadata
I20260809 15:01:38.952773 2574 catalog_manager.cc:4841] TS 9d8564c664b04de3ac0999b4b2c1fb31 (127.27.186.130:34249): tablet c7367180c5f7482b89daf40e15a7cc70 (table test-table [id=abc6b390dc4a430c8d9b658ff5f51715]) successfully deleted
I20260809 15:01:38.956178 3382 ts_tablet_manager.cc:1918] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:38.956341 3382 log.cc:1196] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/c7367180c5f7482b89daf40e15a7cc70
I20260809 15:01:38.956849 3382 ts_tablet_manager.cc:1939] T c7367180c5f7482b89daf40e15a7cc70 P a1d7d822680c4628b1dc6199ba9b511b: Deleting consensus metadata
I20260809 15:01:38.958422 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:38.927182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.27.186.129:45905)
0809 15:01:38.927339 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:38.927344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:38.932236 (+ 4892us) server_negotiation.cc:407] Connection header received
0809 15:01:38.932531 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:38.932538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:38.932622 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:38.932710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:38.939878 (+ 7168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.940971 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.942765 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:38.943452 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:38.943552 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:38.951120 (+ 7568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:38.951134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:38.951150 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:38.951192 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:38.955052 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.955871 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.955879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.955882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.956046 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:38.956512 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:38.956518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:38.956521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:38.956987 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:38.957301 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:38.957608 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 15:01:38.957852 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:01:38.958518 2574 catalog_manager.cc:4841] TS a1d7d822680c4628b1dc6199ba9b511b (127.27.186.129:45905): tablet c7367180c5f7482b89daf40e15a7cc70 (table test-table [id=abc6b390dc4a430c8d9b658ff5f51715]) successfully deleted
I20260809 15:01:38.959008 2886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:39.046543 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 2690
I20260809 15:01:39.071815 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 2902
Aug 09 15:01:39 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:39.086483 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:39.072560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.27.186.130:34249)
0809 15:01:39.072734 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:39.072740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:39.072800 (+ 60us) server_negotiation.cc:407] Connection header received
0809 15:01:39.072995 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:39.073001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:39.073071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:39.073167 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:39.074393 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:39.075130 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:39.076352 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:39.076742 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:39.076853 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:39.081022 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:39.081036 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:39.081051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:39.081089 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:39.083740 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:39.084393 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:39.084401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:39.084403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:39.084532 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:39.084935 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:39.084939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:39.084941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:39.085311 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:39.085595 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:39.085915 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 15:01:39.086161 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:01:39.087113 3101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:39.173633 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 2902
I20260809 15:01:39.208047 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 3116
Aug 09 15:01:39 dist-test-slave-q9h9 krb5kdc[2522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:39.223618 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:39.208940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.27.186.131:36611)
0809 15:01:39.209160 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:39.209165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:39.209180 (+ 15us) server_negotiation.cc:407] Connection header received
0809 15:01:39.209231 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:39.209238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:39.209326 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:39.209411 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:39.210756 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:39.211564 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:39.212788 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:39.213248 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:39.213374 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:39.217773 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:39.217785 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:39.217801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:39.217834 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:39.220893 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:39.221495 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:39.221502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:39.221504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:39.221676 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:39.222073 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:39.222076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:39.222078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:39.222432 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:39.222740 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:39.223032 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 15:01:39.223257 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 15:01:39.224366 3318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:39.343547 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 3116
I20260809 15:01:39.371691 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 2547
I20260809 15:01:39.372771 2639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:39.476979 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 2547
2026-08-09T15:01:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4043 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T15:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:39Z Disabled control of system clock
I20260809 15:01:39.548410 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43689
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 15:01:39.855842 3398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:39.856081 3398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:39.856166 3398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:39.862942 3398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:39.863029 3398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:39.863075 3398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:39.863128 3398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:39.863170 3398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:39.871606 3398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:43689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:39.872573 3398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:39.874162 3398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:39.884814 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:39.884845 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:39.885116 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:39.885603 3398 server_base.cc:1034] running on GCE node
I20260809 15:01:39.886376 3398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:39.887553 3398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:39.888751 3398 hybrid_clock.cc:648] HybridClock initialized: now 1786287699888711 us; error 29 us; skew 500 ppm
I20260809 15:01:39.889058 3398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:39.891471 3398 webserver.cc:460] Webserver started at http://127.27.186.190:37139/ using document root <none> and password file <none>
I20260809 15:01:39.892071 3398 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:39.892170 3398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:39.892485 3398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:39.895145 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "789ee109d7674acfa7d0307c7b6d87b2"
format_stamp: "Formatted at 2026-08-09 15:01:39 on dist-test-slave-q9h9"
server_key: "fcbbf1186cf6d25191141746802a340a"
server_key_iv: "29f81f743bdf60de4a3b36da31ac7feb"
server_key_version: "encryptionkey@0"
I20260809 15:01:39.895874 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "789ee109d7674acfa7d0307c7b6d87b2"
format_stamp: "Formatted at 2026-08-09 15:01:39 on dist-test-slave-q9h9"
server_key: "fcbbf1186cf6d25191141746802a340a"
server_key_iv: "29f81f743bdf60de4a3b36da31ac7feb"
server_key_version: "encryptionkey@0"
I20260809 15:01:39.900430 3398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:39.903492 3418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:39.904599 3398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:39.904739 3398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "789ee109d7674acfa7d0307c7b6d87b2"
format_stamp: "Formatted at 2026-08-09 15:01:39 on dist-test-slave-q9h9"
server_key: "fcbbf1186cf6d25191141746802a340a"
server_key_iv: "29f81f743bdf60de4a3b36da31ac7feb"
server_key_version: "encryptionkey@0"
I20260809 15:01:39.905053 3398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:39.910213 3398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:39.910889 3398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:39.911160 3398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:39.927019 3398 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:43689
I20260809 15:01:39.927057 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:43689 every 8 connection(s)
I20260809 15:01:39.928115 3398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:39.931361 3513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:39.933779 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 3398
I20260809 15:01:39.933967 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:39.934779 28394 external_mini_cluster.cc:1428] Setting key d691db3246dcf87bbb3e3d6caa001e20
I20260809 15:01:39.943553 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:39.936751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.27.186.190:43689)
0809 15:01:39.937716 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:39.937729 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:39.937772 (+ 43us) server_negotiation.cc:407] Connection header received
0809 15:01:39.939918 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:39.939945 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:39.940445 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:39.940847 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:39.941789 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:39.941797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:39.941812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:39.941829 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:39.941894 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:39.941918 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:39.942156 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:39.942303 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 15:01:39.942527 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":302,"threads_started":1}
I20260809 15:01:39.944303 3513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2: Bootstrap starting.
I20260809 15:01:39.947350 3513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:39.948274 3513 log.cc:824] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:39.950757 3513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2: No bootstrap required, opened a new log
I20260809 15:01:39.957242 3513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789ee109d7674acfa7d0307c7b6d87b2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43689 } }
I20260809 15:01:39.957592 3513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:39.957679 3513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789ee109d7674acfa7d0307c7b6d87b2, State: Initialized, Role: FOLLOWER
I20260809 15:01:39.958231 3513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789ee109d7674acfa7d0307c7b6d87b2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43689 } }
I20260809 15:01:39.958442 3513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:39.958566 3513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:39.958704 3513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:39.960568 3513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789ee109d7674acfa7d0307c7b6d87b2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43689 } }
I20260809 15:01:39.961015 3513 leader_election.cc:302] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 789ee109d7674acfa7d0307c7b6d87b2; no voters:
I20260809 15:01:39.961624 3513 leader_election.cc:288] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:39.961712 3519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:39.962378 3519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [term 1 LEADER]: Becoming Leader. State: Replica: 789ee109d7674acfa7d0307c7b6d87b2, State: Running, Role: LEADER
I20260809 15:01:39.962893 3519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789ee109d7674acfa7d0307c7b6d87b2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43689 } }
I20260809 15:01:39.963409 3513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:39.965235 3521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789ee109d7674acfa7d0307c7b6d87b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789ee109d7674acfa7d0307c7b6d87b2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43689 } } }
I20260809 15:01:39.965586 3521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:39.965756 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789ee109d7674acfa7d0307c7b6d87b2. Latest consensus state: current_term: 1 leader_uuid: "789ee109d7674acfa7d0307c7b6d87b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789ee109d7674acfa7d0307c7b6d87b2" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43689 } } }
I20260809 15:01:39.966120 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:39.966698 3533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:39.972025 3533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:39.981105 3533 catalog_manager.cc:1347] Generated new cluster ID: a79776c9bdcd4eff89be7d1235620dda
I20260809 15:01:39.981200 3533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:40.025441 3533 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:40.025898 3533 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:40.041293 3533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2: Generated new TSK 0
I20260809 15:01:40.042214 3533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:01:40.056036 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43689
--builtin_ntp_servers=127.27.186.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 15:01:40.379364 3554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:40.379606 3554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:40.379714 3554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:40.386699 3554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:40.386790 3554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:40.386936 3554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:40.395722 3554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:40.396828 3554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:40.398530 3554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:40.410555 3561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:40.410604 3562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:40.410643 3564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:40.411574 3554 server_base.cc:1034] running on GCE node
I20260809 15:01:40.412147 3554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:40.412992 3554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:40.414191 3554 hybrid_clock.cc:648] HybridClock initialized: now 1786287700414142 us; error 36 us; skew 500 ppm
I20260809 15:01:40.414471 3554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:40.416487 3554 webserver.cc:460] Webserver started at http://127.27.186.129:34063/ using document root <none> and password file <none>
I20260809 15:01:40.417097 3554 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:40.417207 3554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:40.417500 3554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:40.420172 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "ee4a8221f6fb43949023a95540365128"
format_stamp: "Formatted at 2026-08-09 15:01:40 on dist-test-slave-q9h9"
server_key: "c96ce0b927f11c222616b4ed7519b897"
server_key_iv: "17176ce36b80c8e8794f7075f6df6330"
server_key_version: "encryptionkey@0"
I20260809 15:01:40.420912 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4a8221f6fb43949023a95540365128"
format_stamp: "Formatted at 2026-08-09 15:01:40 on dist-test-slave-q9h9"
server_key: "c96ce0b927f11c222616b4ed7519b897"
server_key_iv: "17176ce36b80c8e8794f7075f6df6330"
server_key_version: "encryptionkey@0"
I20260809 15:01:40.425621 3554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260809 15:01:40.428843 3571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:40.429984 3554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:40.430168 3554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "ee4a8221f6fb43949023a95540365128"
format_stamp: "Formatted at 2026-08-09 15:01:40 on dist-test-slave-q9h9"
server_key: "c96ce0b927f11c222616b4ed7519b897"
server_key_iv: "17176ce36b80c8e8794f7075f6df6330"
server_key_version: "encryptionkey@0"
I20260809 15:01:40.430344 3554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:40.435456 3554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:40.436132 3554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:40.436342 3554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:40.437134 3554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:40.438400 3554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:40.438486 3554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:40.438565 3554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:40.438618 3554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:40.467315 3554 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:45933
I20260809 15:01:40.467363 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:45933 every 8 connection(s)
I20260809 15:01:40.468509 3554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:40.475875 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:40.470918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:35723 (local address 127.27.186.190:43689)
0809 15:01:40.471340 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:40.471344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:40.472425 (+ 1081us) server_negotiation.cc:407] Connection header received
0809 15:01:40.474006 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:40.474010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:40.474092 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:40.474191 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:40.475310 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:40.475314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:40.475317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:40.475332 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:40.475371 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:40.475382 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:40.475470 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:40.475595 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 15:01:40.475643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260809 15:01:40.476266 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 3554
I20260809 15:01:40.476430 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:40.476717 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:40.471441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43689 (local address 127.27.186.129:35723)
0809 15:01:40.472278 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:40.472321 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:40.473779 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:40.474382 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:40.474389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:40.474473 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:40.475043 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:40.475045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:40.475078 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:40.475156 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:40.475453 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:40.475458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:40.475463 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:01:40.475651 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 15:01:40.475849 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":248,"threads_started":1}
I20260809 15:01:40.477065 28394 external_mini_cluster.cc:1428] Setting key e346ca930ddb36080c3c9ec75f3392bd
I20260809 15:01:40.478272 3761 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43689
I20260809 15:01:40.478524 3761 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:40.479183 3761 heartbeater.cc:511] Master 127.27.186.190:43689 requested a full tablet report, sending...
I20260809 15:01:40.480110 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43689
--builtin_ntp_servers=127.27.186.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 15:01:40.481439 3448 ts_manager.cc:194] Registered new tserver with Master: ee4a8221f6fb43949023a95540365128 (127.27.186.129:45933)
W20260809 15:01:40.790201 3769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:40.790449 3769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:40.790575 3769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:40.797448 3769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:40.797539 3769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:40.797685 3769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:40.806176 3769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:40.807206 3769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:40.808789 3769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:40.820467 3777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:40.820550 3775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:40.820559 3774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:40.821784 3769 server_base.cc:1034] running on GCE node
I20260809 15:01:40.822403 3769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:40.823230 3769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:40.824424 3769 hybrid_clock.cc:648] HybridClock initialized: now 1786287700824374 us; error 45 us; skew 500 ppm
I20260809 15:01:40.824715 3769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:40.826764 3769 webserver.cc:460] Webserver started at http://127.27.186.130:34863/ using document root <none> and password file <none>
I20260809 15:01:40.827361 3769 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:40.827461 3769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:40.827764 3769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:40.830334 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "df12bf408c6240b8926dbdd5a4075bb1"
format_stamp: "Formatted at 2026-08-09 15:01:40 on dist-test-slave-q9h9"
server_key: "a90863a1de792447c9405f512d9f11b2"
server_key_iv: "66f6a7141dbd3cae2eb049692efb5613"
server_key_version: "encryptionkey@0"
I20260809 15:01:40.831148 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "df12bf408c6240b8926dbdd5a4075bb1"
format_stamp: "Formatted at 2026-08-09 15:01:40 on dist-test-slave-q9h9"
server_key: "a90863a1de792447c9405f512d9f11b2"
server_key_iv: "66f6a7141dbd3cae2eb049692efb5613"
server_key_version: "encryptionkey@0"
I20260809 15:01:40.835976 3769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 15:01:40.839113 3792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:40.840233 3769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 15:01:40.840376 3769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "df12bf408c6240b8926dbdd5a4075bb1"
format_stamp: "Formatted at 2026-08-09 15:01:40 on dist-test-slave-q9h9"
server_key: "a90863a1de792447c9405f512d9f11b2"
server_key_iv: "66f6a7141dbd3cae2eb049692efb5613"
server_key_version: "encryptionkey@0"
I20260809 15:01:40.840561 3769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:40.845715 3769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:40.846463 3769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:40.846707 3769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:40.847522 3769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:40.848914 3769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:40.849009 3769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:40.849113 3769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:40.849185 3769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:40.878566 3769 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:39627
I20260809 15:01:40.878641 3973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:39627 every 8 connection(s)
I20260809 15:01:40.879696 3769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:40.887246 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:40.881970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:43421 (local address 127.27.186.190:43689)
0809 15:01:40.882223 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:40.882228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:40.883450 (+ 1222us) server_negotiation.cc:407] Connection header received
0809 15:01:40.885106 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:40.885113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:40.885183 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:40.885270 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:40.886520 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:40.886527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:40.886531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:40.886548 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:40.886579 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:40.886591 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:40.886697 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:40.886872 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 15:01:40.886938 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 15:01:40.888003 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:40.882457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43689 (local address 127.27.186.130:43421)
0809 15:01:40.883306 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:40.883340 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:40.884840 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:40.885500 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:40.885508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:40.885590 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:40.886246 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:40.886248 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:40.886283 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:40.886363 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:40.886680 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:40.886691 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:40.886698 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:01:40.886939 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 15:01:40.887160 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
I20260809 15:01:40.889137 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 3769
I20260809 15:01:40.889228 3974 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43689
I20260809 15:01:40.889281 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:40.889490 3974 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:40.889708 28394 external_mini_cluster.cc:1428] Setting key 8322498bf4530e6de36a757b07b53b98
I20260809 15:01:40.890059 3974 heartbeater.cc:511] Master 127.27.186.190:43689 requested a full tablet report, sending...
I20260809 15:01:40.891417 3447 ts_manager.cc:194] Registered new tserver with Master: df12bf408c6240b8926dbdd5a4075bb1 (127.27.186.130:39627)
I20260809 15:01:40.892449 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43689
--builtin_ntp_servers=127.27.186.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 15:01:41.199473 3978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:41.199717 3978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:41.199844 3978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:41.207062 3978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:41.207151 3978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:41.207299 3978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:41.215925 3978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:41.216976 3978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:41.218606 3978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:41.230644 3986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:41.230589 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:41.230621 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:41.231504 3978 server_base.cc:1034] running on GCE node
I20260809 15:01:41.232043 3978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:41.232935 3978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:41.234115 3978 hybrid_clock.cc:648] HybridClock initialized: now 1786287701234051 us; error 53 us; skew 500 ppm
I20260809 15:01:41.234440 3978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:41.236569 3978 webserver.cc:460] Webserver started at http://127.27.186.131:34599/ using document root <none> and password file <none>
I20260809 15:01:41.237208 3978 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:41.237308 3978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:41.237612 3978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:41.240491 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "81a478c772284c2795a6ec7cb517dfb4"
format_stamp: "Formatted at 2026-08-09 15:01:41 on dist-test-slave-q9h9"
server_key: "e300719a54b76dfd4da1520cadba53e6"
server_key_iv: "bc9a58fb8cbb60cec4037f72688ecb2d"
server_key_version: "encryptionkey@0"
I20260809 15:01:41.241248 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "81a478c772284c2795a6ec7cb517dfb4"
format_stamp: "Formatted at 2026-08-09 15:01:41 on dist-test-slave-q9h9"
server_key: "e300719a54b76dfd4da1520cadba53e6"
server_key_iv: "bc9a58fb8cbb60cec4037f72688ecb2d"
server_key_version: "encryptionkey@0"
I20260809 15:01:41.246193 3978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 15:01:41.249357 4002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:41.250641 3978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:41.250787 3978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "81a478c772284c2795a6ec7cb517dfb4"
format_stamp: "Formatted at 2026-08-09 15:01:41 on dist-test-slave-q9h9"
server_key: "e300719a54b76dfd4da1520cadba53e6"
server_key_iv: "bc9a58fb8cbb60cec4037f72688ecb2d"
server_key_version: "encryptionkey@0"
I20260809 15:01:41.250973 3978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:41.256081 3978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:41.256786 3978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:41.257025 3978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:41.257846 3978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:41.259174 3978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:41.259268 3978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:41.259377 3978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:41.259445 3978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:41.288254 3978 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:34077
I20260809 15:01:41.288323 4170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:34077 every 8 connection(s)
I20260809 15:01:41.289427 3978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:41.290678 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 3978
I20260809 15:01:41.290833 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:41.291280 28394 external_mini_cluster.cc:1428] Setting key c92a5bb07e9d47d7678b7826879079cc
I20260809 15:01:41.297037 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.291790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:33549 (local address 127.27.186.190:43689)
0809 15:01:41.292005 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.292012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.293464 (+ 1452us) server_negotiation.cc:407] Connection header received
0809 15:01:41.295246 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.295250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.295304 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.295382 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.296494 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.296499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.296502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.296518 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.296549 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.296560 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.296655 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.296791 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.296837 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 15:01:41.297812 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.292304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43689 (local address 127.27.186.131:33549)
0809 15:01:41.293315 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.293357 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:41.295009 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:41.295584 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:41.295591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:41.295674 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:41.296241 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:41.296243 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:41.296275 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:41.296350 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:41.296630 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:41.296635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:41.296640 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:01:41.296839 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 15:01:41.297009 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":304,"threads_started":1}
I20260809 15:01:41.299026 4173 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43689
I20260809 15:01:41.299254 4173 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:41.299744 4173 heartbeater.cc:511] Master 127.27.186.190:43689 requested a full tablet report, sending...
I20260809 15:01:41.300879 3447 ts_manager.cc:194] Registered new tserver with Master: 81a478c772284c2795a6ec7cb517dfb4 (127.27.186.131:34077)
I20260809 15:01:41.306227 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:01:41.312361 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.310369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.27.186.190:43689)
0809 15:01:41.310513 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.310517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.310941 (+ 424us) server_negotiation.cc:407] Connection header received
0809 15:01:41.311061 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.311064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.311107 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.311178 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.311931 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.311935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.311938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.311952 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.311980 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.311991 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.312079 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.312128 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.312168 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:01:41.315850 3447 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:41.322302 3447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:41.339124 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.333037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47346)
0809 15:01:41.335029 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.335048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:41.335149 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:41.337003 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:41.337010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:41.337084 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:41.337920 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:41.337924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:41.337950 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:41.338067 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:41.338440 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:41.338452 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:41.338459 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:01:41.338598 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 15:01:41.338683 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1840,"thread_start_us":387,"threads_started":1}
I20260809 15:01:41.339515 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.333554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.27.186.129:45933)
0809 15:01:41.336503 (+ 2949us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.336512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.336539 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:01:41.336607 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.336613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.336821 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.336969 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.338194 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.338202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.338216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.338234 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.338290 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.338316 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.339107 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.339172 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.339284 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2794,"thread_start_us":2670,"threads_started":1}
I20260809 15:01:41.339777 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.333144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.27.186.130:39627)
0809 15:01:41.335709 (+ 2565us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.335727 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.335762 (+ 35us) server_negotiation.cc:407] Connection header received
0809 15:01:41.335832 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.335840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.336056 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.336203 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.338303 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.338311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.338324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.338352 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.338411 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.338434 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.338610 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.339337 (+ 727us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.339435 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2409,"thread_start_us":2256,"threads_started":1}
I20260809 15:01:41.339833 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.332859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39627 (local address 127.0.0.1:51794)
0809 15:01:41.333673 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.333717 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:41.333837 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:41.337319 (+ 3482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:41.337324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:41.337352 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:41.338079 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:41.338082 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:41.338098 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:41.338140 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:41.338667 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:41.338672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:41.338677 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:01:41.338824 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 15:01:41.338877 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":1850,"threads_started":1}
I20260809 15:01:41.341591 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.335274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.27.186.131:34077)
0809 15:01:41.336820 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.336825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.336849 (+ 24us) server_negotiation.cc:407] Connection header received
0809 15:01:41.336901 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.336907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.337083 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.337221 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.340852 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.340860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.340873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.340887 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.340934 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.340947 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.341091 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.341151 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.341241 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":1274,"threads_started":1}
I20260809 15:01:41.341619 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.334741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34077 (local address 127.0.0.1:50288)
0809 15:01:41.335957 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.335972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:41.336055 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:41.337580 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:41.337583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:41.337605 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:41.340639 (+ 3034us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:41.340643 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:41.340661 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:41.340703 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:41.341039 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:41.341043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:41.341047 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 15:01:41.341163 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 15:01:41.341212 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":979,"threads_started":1}
I20260809 15:01:41.343132 3653 tablet_service.cc:1467] Processing CreateTablet for tablet 1972410bb294416b9e986ace7f82f2d0 (DEFAULT_TABLE table=test-table [id=e1d9525b3bb94e959d74a7446fff956d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:41.343132 3866 tablet_service.cc:1467] Processing CreateTablet for tablet 1972410bb294416b9e986ace7f82f2d0 (DEFAULT_TABLE table=test-table [id=e1d9525b3bb94e959d74a7446fff956d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:41.344552 3653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1972410bb294416b9e986ace7f82f2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:41.344558 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1972410bb294416b9e986ace7f82f2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:41.344856 4070 tablet_service.cc:1467] Processing CreateTablet for tablet 1972410bb294416b9e986ace7f82f2d0 (DEFAULT_TABLE table=test-table [id=e1d9525b3bb94e959d74a7446fff956d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:41.345870 4070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1972410bb294416b9e986ace7f82f2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:41.356419 4199 tablet_bootstrap.cc:492] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Bootstrap starting.
I20260809 15:01:41.358486 4203 tablet_bootstrap.cc:492] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: Bootstrap starting.
I20260809 15:01:41.360766 4199 tablet_bootstrap.cc:654] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:41.361881 4203 tablet_bootstrap.cc:654] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:41.361975 4199 log.cc:824] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:41.362673 4200 tablet_bootstrap.cc:492] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Bootstrap starting.
I20260809 15:01:41.363284 4203 log.cc:824] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:41.365988 4199 tablet_bootstrap.cc:492] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: No bootstrap required, opened a new log
I20260809 15:01:41.366065 4203 tablet_bootstrap.cc:492] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: No bootstrap required, opened a new log
I20260809 15:01:41.366281 4200 tablet_bootstrap.cc:654] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:41.366278 4199 ts_tablet_manager.cc:1397] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260809 15:01:41.366307 4203 ts_tablet_manager.cc:1397] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:01:41.367408 4200 log.cc:824] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:41.370252 4200 tablet_bootstrap.cc:492] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: No bootstrap required, opened a new log
I20260809 15:01:41.370479 4200 ts_tablet_manager.cc:1397] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:01:41.375658 4203 raft_consensus.cc:348] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.375991 4203 raft_consensus.cc:374] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:41.375840 4199 raft_consensus.cc:348] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.376085 4203 raft_consensus.cc:729] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4a8221f6fb43949023a95540365128, State: Initialized, Role: FOLLOWER
I20260809 15:01:41.376223 4199 raft_consensus.cc:374] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:41.376325 4199 raft_consensus.cc:729] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a478c772284c2795a6ec7cb517dfb4, State: Initialized, Role: FOLLOWER
I20260809 15:01:41.376657 4203 consensus_queue.cc:260] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.376900 4199 consensus_queue.cc:260] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.377722 3761 heartbeater.cc:503] Master 127.27.186.190:43689 was elected leader, sending a full tablet report...
I20260809 15:01:41.378377 4203 ts_tablet_manager.cc:1428] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260809 15:01:41.379288 4173 heartbeater.cc:503] Master 127.27.186.190:43689 was elected leader, sending a full tablet report...
I20260809 15:01:41.379904 4199 ts_tablet_manager.cc:1428] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260809 15:01:41.380614 4200 raft_consensus.cc:348] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.381013 4200 raft_consensus.cc:374] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:41.381124 4200 raft_consensus.cc:729] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df12bf408c6240b8926dbdd5a4075bb1, State: Initialized, Role: FOLLOWER
I20260809 15:01:41.381901 4200 consensus_queue.cc:260] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.383072 3974 heartbeater.cc:503] Master 127.27.186.190:43689 was elected leader, sending a full tablet report...
I20260809 15:01:41.383471 4200 ts_tablet_manager.cc:1428] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260809 15:01:41.416849 4211 raft_consensus.cc:482] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:41.417073 4211 raft_consensus.cc:504] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.418553 4211 leader_election.cc:288] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df12bf408c6240b8926dbdd5a4075bb1 (127.27.186.130:39627), 81a478c772284c2795a6ec7cb517dfb4 (127.27.186.131:34077)
I20260809 15:01:41.421473 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.418868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:44327 (local address 127.27.186.130:39627)
0809 15:01:41.419037 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.419043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.419768 (+ 725us) server_negotiation.cc:407] Connection header received
0809 15:01:41.419908 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.419911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.419956 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.420031 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.420906 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.420912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.420917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.420933 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.420963 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.420974 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.421076 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.421169 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.421237 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:01:41.421617 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.419063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:52619 (local address 127.27.186.131:34077)
0809 15:01:41.419234 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.419240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.419855 (+ 615us) server_negotiation.cc:407] Connection header received
0809 15:01:41.419995 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.419998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.420041 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.420115 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.421062 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.421066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.421069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.421085 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.421113 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.421123 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.421227 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.421311 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.421367 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:01:41.421957 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.419009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34077 (local address 127.27.186.129:52619)
0809 15:01:41.419745 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.419762 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:41.419879 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:41.420151 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:41.420160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:41.420185 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:41.420880 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:41.420882 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:41.420895 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:41.420931 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:41.421210 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:41.421214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:41.421219 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:01:41.421352 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 15:01:41.421404 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":406,"threads_started":1}
I20260809 15:01:41.421985 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.418929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39627 (local address 127.27.186.129:44327)
0809 15:01:41.419659 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.419679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:41.419783 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:41.420069 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:41.420074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:41.420096 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:41.420725 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:01:41.420728 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:41.420742 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:01:41.420778 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:41.421055 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:41.421060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:41.421065 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:01:41.421209 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 15:01:41.421257 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":403,"threads_started":1}
I20260809 15:01:41.422847 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1972410bb294416b9e986ace7f82f2d0" candidate_uuid: "ee4a8221f6fb43949023a95540365128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df12bf408c6240b8926dbdd5a4075bb1" is_pre_election: true
I20260809 15:01:41.422914 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1972410bb294416b9e986ace7f82f2d0" candidate_uuid: "ee4a8221f6fb43949023a95540365128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a478c772284c2795a6ec7cb517dfb4" is_pre_election: true
I20260809 15:01:41.423228 3902 raft_consensus.cc:2454] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4a8221f6fb43949023a95540365128 in term 0.
I20260809 15:01:41.423327 4099 raft_consensus.cc:2454] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4a8221f6fb43949023a95540365128 in term 0.
I20260809 15:01:41.423774 3573 leader_election.cc:302] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df12bf408c6240b8926dbdd5a4075bb1, ee4a8221f6fb43949023a95540365128; no voters:
I20260809 15:01:41.424075 4211 raft_consensus.cc:2781] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:41.424201 4211 raft_consensus.cc:482] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:41.424296 4211 raft_consensus.cc:3037] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:41.426415 4211 raft_consensus.cc:504] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.426921 4211 leader_election.cc:288] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [CANDIDATE]: Term 1 election: Requested vote from peers df12bf408c6240b8926dbdd5a4075bb1 (127.27.186.130:39627), 81a478c772284c2795a6ec7cb517dfb4 (127.27.186.131:34077)
I20260809 15:01:41.427351 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1972410bb294416b9e986ace7f82f2d0" candidate_uuid: "ee4a8221f6fb43949023a95540365128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a478c772284c2795a6ec7cb517dfb4"
I20260809 15:01:41.427351 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1972410bb294416b9e986ace7f82f2d0" candidate_uuid: "ee4a8221f6fb43949023a95540365128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df12bf408c6240b8926dbdd5a4075bb1"
I20260809 15:01:41.427537 4099 raft_consensus.cc:3037] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:41.427538 3902 raft_consensus.cc:3037] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:41.430572 3902 raft_consensus.cc:2454] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4a8221f6fb43949023a95540365128 in term 1.
I20260809 15:01:41.430572 4099 raft_consensus.cc:2454] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4a8221f6fb43949023a95540365128 in term 1.
I20260809 15:01:41.431125 3576 leader_election.cc:302] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a478c772284c2795a6ec7cb517dfb4, ee4a8221f6fb43949023a95540365128; no voters:
I20260809 15:01:41.431422 4211 raft_consensus.cc:2781] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:41.431813 4211 raft_consensus.cc:686] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 1 LEADER]: Becoming Leader. State: Replica: ee4a8221f6fb43949023a95540365128, State: Running, Role: LEADER
I20260809 15:01:41.432248 4211 consensus_queue.cc:237] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } }
I20260809 15:01:41.436179 3448 catalog_manager.cc:5495] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee4a8221f6fb43949023a95540365128 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "ee4a8221f6fb43949023a95540365128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4a8221f6fb43949023a95540365128" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 } health_report { overall_health: UNKNOWN } } }
I20260809 15:01:41.462211 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.460242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.27.186.129:45933)
0809 15:01:41.460408 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.460413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.460430 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:01:41.460641 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.460646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.460719 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.460807 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.461623 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.461629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.461634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.461652 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.461681 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.461692 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.461795 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.461889 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.461942 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:01:41.469627 4099 raft_consensus.cc:1261] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 1 FOLLOWER]: Refusing update from remote peer ee4a8221f6fb43949023a95540365128: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:41.469627 3902 raft_consensus.cc:1261] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 1 FOLLOWER]: Refusing update from remote peer ee4a8221f6fb43949023a95540365128: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:01:41.470432 3762 tablet.cc:2369] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:41.470602 4216 consensus_queue.cc:1035] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df12bf408c6240b8926dbdd5a4075bb1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:41.470928 4211 consensus_queue.cc:1035] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a478c772284c2795a6ec7cb517dfb4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:41.486153 4221 mvcc.cc:205] Tried to move back new op lower bound from 7316634425210273792 to 7316634425069891584. Current Snapshot: MvccSnapshot[applied={T|T < 7316634425210273792}]
I20260809 15:01:41.494398 4226 mvcc.cc:205] Tried to move back new op lower bound from 7316634425210273792 to 7316634425069891584. Current Snapshot: MvccSnapshot[applied={T|T < 7316634425210273792}]
I20260809 15:01:41.499529 3448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:41.499786 3448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:41.503202 3448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 789ee109d7674acfa7d0307c7b6d87b2: Sending DeleteTablet for 3 replicas of tablet 1972410bb294416b9e986ace7f82f2d0
I20260809 15:01:41.504230 3866 tablet_service.cc:1514] Processing DeleteTablet for tablet 1972410bb294416b9e986ace7f82f2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:41 UTC) from {username='slave'} at 127.0.0.1:51794
I20260809 15:01:41.504479 4070 tablet_service.cc:1514] Processing DeleteTablet for tablet 1972410bb294416b9e986ace7f82f2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:41 UTC) from {username='slave'} at 127.0.0.1:50288
I20260809 15:01:41.504675 3653 tablet_service.cc:1514] Processing DeleteTablet for tablet 1972410bb294416b9e986ace7f82f2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:41 UTC) from {username='slave'} at 127.0.0.1:47346
I20260809 15:01:41.509377 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 3554
I20260809 15:01:41.519515 4237 tablet_replica.cc:330] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: stopping tablet replica
I20260809 15:01:41.519922 4237 raft_consensus.cc:2229] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:41.520392 4237 raft_consensus.cc:2258] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:41.522382 4237 ts_tablet_manager.cc:1905] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:41.529110 4237 ts_tablet_manager.cc:1918] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:41.529258 4237 log.cc:1196] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/1972410bb294416b9e986ace7f82f2d0
I20260809 15:01:41.529790 4237 ts_tablet_manager.cc:1939] T 1972410bb294416b9e986ace7f82f2d0 P df12bf408c6240b8926dbdd5a4075bb1: Deleting consensus metadata
I20260809 15:01:41.531512 4239 tablet_replica.cc:330] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: stopping tablet replica
I20260809 15:01:41.531774 3423 catalog_manager.cc:4841] TS df12bf408c6240b8926dbdd5a4075bb1 (127.27.186.130:39627): tablet 1972410bb294416b9e986ace7f82f2d0 (table test-table [id=e1d9525b3bb94e959d74a7446fff956d]) successfully deleted
I20260809 15:01:41.532642 4240 tablet_replica.cc:330] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128: stopping tablet replica
I20260809 15:01:41.532873 4239 raft_consensus.cc:2229] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:41.533082 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.510454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.27.186.129:45933)
0809 15:01:41.510618 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.510625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.529799 (+ 19174us) server_negotiation.cc:407] Connection header received
0809 15:01:41.530373 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.530379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.530445 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.530518 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.532188 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.532193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.532202 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.532224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.532266 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.532293 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.532421 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.532666 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.532748 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 15:01:41.533089 4240 raft_consensus.cc:2229] T 1972410bb294416b9e986ace7f82f2d0 P ee4a8221f6fb43949023a95540365128 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:41.533825 4239 raft_consensus.cc:2258] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:41.533985 3751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:41.535935 4239 ts_tablet_manager.cc:1905] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:41.541333 4239 ts_tablet_manager.cc:1918] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:41.541458 4239 log.cc:1196] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/1972410bb294416b9e986ace7f82f2d0
I20260809 15:01:41.541857 4239 ts_tablet_manager.cc:1939] T 1972410bb294416b9e986ace7f82f2d0 P 81a478c772284c2795a6ec7cb517dfb4: Deleting consensus metadata
I20260809 15:01:41.543208 3425 catalog_manager.cc:4841] TS 81a478c772284c2795a6ec7cb517dfb4 (127.27.186.131:34077): tablet 1972410bb294416b9e986ace7f82f2d0 (table test-table [id=e1d9525b3bb94e959d74a7446fff956d]) successfully deleted
I20260809 15:01:41.629124 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 3554
I20260809 15:01:41.656227 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 3769
W20260809 15:01:41.656188 3424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.186.129:45933 (error 108)
I20260809 15:01:41.658337 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.657709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47376)
0809 15:01:41.657907 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.658146 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 15:01:41.658558 3424 catalog_manager.cc:4584] TS ee4a8221f6fb43949023a95540365128 (127.27.186.129:45933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1972410bb294416b9e986ace7f82f2d0: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
I20260809 15:01:41.659090 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.656957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.27.186.130:39627)
0809 15:01:41.657143 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.657148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.657171 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:01:41.657318 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.657325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.657417 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.657505 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.658457 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.658464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.658468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.658490 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.658530 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.658546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.658680 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.658798 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.658856 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:01:41.659758 3968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:41.712272 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.711744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47384)
0809 15:01:41.711970 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.712091 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 15:01:41.713264 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.712889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47396)
0809 15:01:41.713044 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.713131 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 15:01:41.745134 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 3769
I20260809 15:01:41.771680 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 3978
I20260809 15:01:41.773567 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.773084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47398)
0809 15:01:41.773276 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.773396 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 15:01:41.774457 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.772493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.27.186.131:34077)
0809 15:01:41.772694 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:41.772700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:41.772723 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:01:41.772790 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:41.772795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:41.772897 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:41.773002 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:41.773847 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:41.773851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:41.773854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:41.773870 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:41.773904 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:41.773916 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:41.774045 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:41.774146 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 15:01:41.774220 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:01:41.774549 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.774169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47400)
0809 15:01:41.774320 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.774405 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 15:01:41.775141 4165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:41.855057 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.854516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47410)
0809 15:01:41.854742 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.854876 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 15:01:41.856174 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:41.855728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45933 (local address 127.0.0.1:47422)
0809 15:01:41.855948 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:41.856032 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 15:01:41.861126 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 3978
I20260809 15:01:41.881414 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 3398
I20260809 15:01:41.882472 3501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:41.984426 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 3398
2026-08-09T15:01:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2508 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 15:01:42.031627 28394 mini_oidc.cc:152] Starting JWKS server
I20260809 15:01:42.034457 28394 webserver.cc:460] Webserver started at https://127.0.0.1:46237/ using document root <none> and password file <none>
I20260809 15:01:42.034783 28394 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 15:01:42.035362 28394 webserver.cc:460] Webserver started at http://127.0.0.1:43421/ using document root <none> and password file <none>
2026-08-09T15:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:42Z Disabled control of system clock
I20260809 15:01:42.062538 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44847
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:44847
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46237/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/testchainca.cert with env {}
W20260809 15:01:42.365721 4259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:42.365984 4259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:42.366127 4259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:42.373024 4259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:42.373147 4259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:42.373210 4259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:42.373288 4259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:42.373348 4259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 15:01:42.373436 4259 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 15:01:42.373492 4259 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46237/jwks.json
I20260809 15:01:42.383539 4259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:44847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46237/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:42.384665 4259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:42.385849 4259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:42.396741 4264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:42.396780 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:42.396727 4274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:42.397692 4259 server_base.cc:1034] running on GCE node
I20260809 15:01:42.398478 4259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:42.399688 4259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:42.400882 4259 hybrid_clock.cc:648] HybridClock initialized: now 1786287702400833 us; error 54 us; skew 500 ppm
I20260809 15:01:42.401196 4259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:42.403224 4259 webserver.cc:460] Webserver started at http://127.27.186.190:35821/ using document root <none> and password file <none>
I20260809 15:01:42.403829 4259 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:42.403932 4259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:42.404210 4259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:42.406883 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "9f9e722f98b6473eb98fcabbabf70c26"
format_stamp: "Formatted at 2026-08-09 15:01:42 on dist-test-slave-q9h9"
server_key: "c8bf343a8955c48e5b95c6aaba7c9556"
server_key_iv: "d63ec827c0244576a8c8e10df8fdb4a5"
server_key_version: "encryptionkey@0"
I20260809 15:01:42.407599 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "9f9e722f98b6473eb98fcabbabf70c26"
format_stamp: "Formatted at 2026-08-09 15:01:42 on dist-test-slave-q9h9"
server_key: "c8bf343a8955c48e5b95c6aaba7c9556"
server_key_iv: "d63ec827c0244576a8c8e10df8fdb4a5"
server_key_version: "encryptionkey@0"
I20260809 15:01:42.412446 4259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260809 15:01:42.415551 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:42.416725 4259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:42.416879 4259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "9f9e722f98b6473eb98fcabbabf70c26"
format_stamp: "Formatted at 2026-08-09 15:01:42 on dist-test-slave-q9h9"
server_key: "c8bf343a8955c48e5b95c6aaba7c9556"
server_key_iv: "d63ec827c0244576a8c8e10df8fdb4a5"
server_key_version: "encryptionkey@0"
I20260809 15:01:42.417058 4259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:42.459841 4259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:42.460592 4259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:42.460866 4259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:42.476948 4259 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:44847
I20260809 15:01:42.476984 4360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:44847 every 8 connection(s)
I20260809 15:01:42.478128 4259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:42.481401 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:42.486819 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 4259
I20260809 15:01:42.487035 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:42.487833 28394 external_mini_cluster.cc:1428] Setting key e2951e10a37feea471bfec809056bf7c
I20260809 15:01:42.492491 4361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26: Bootstrap starting.
I20260809 15:01:42.495431 4361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:42.496353 4361 log.cc:824] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:42.499135 4361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26: No bootstrap required, opened a new log
I20260809 15:01:42.500404 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:42.489686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.27.186.190:44847)
0809 15:01:42.490671 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:42.490691 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:42.490732 (+ 41us) server_negotiation.cc:407] Connection header received
0809 15:01:42.492323 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:42.492343 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:42.492759 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:42.493084 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:42.493661 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:42.494958 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:42.496283 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:42.497088 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:42.497294 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:42.498339 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:42.498346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:42.498358 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:42.498843 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:42.498917 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:42.498937 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:42.499024 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:42.499286 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 15:01:42.499563 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":376,"threads_started":1}
I20260809 15:01:42.503875 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:01:42.507359 4361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9e722f98b6473eb98fcabbabf70c26" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44847 } }
I20260809 15:01:42.507653 4361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:42.507740 4361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9e722f98b6473eb98fcabbabf70c26, State: Initialized, Role: FOLLOWER
I20260809 15:01:42.508204 4361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9e722f98b6473eb98fcabbabf70c26" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44847 } }
I20260809 15:01:42.508410 4361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:42.508526 4361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:42.508672 4361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:42.510691 4361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9e722f98b6473eb98fcabbabf70c26" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44847 } }
I20260809 15:01:42.511126 4361 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9e722f98b6473eb98fcabbabf70c26; no voters:
I20260809 15:01:42.511616 4361 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:42.511718 4366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:42.512724 4366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9e722f98b6473eb98fcabbabf70c26, State: Running, Role: LEADER
I20260809 15:01:42.513221 4366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9e722f98b6473eb98fcabbabf70c26" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44847 } }
I20260809 15:01:42.513470 4361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:42.515923 4369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9e722f98b6473eb98fcabbabf70c26. Latest consensus state: current_term: 1 leader_uuid: "9f9e722f98b6473eb98fcabbabf70c26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9e722f98b6473eb98fcabbabf70c26" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44847 } } }
I20260809 15:01:42.516294 4369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:42.516968 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f9e722f98b6473eb98fcabbabf70c26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9e722f98b6473eb98fcabbabf70c26" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44847 } } }
I20260809 15:01:42.517268 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26 [sys.catalog]: This master's current role is: LEADER
W20260809 15:01:42.523021 4390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:01:42.523124 4390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:01:42.523262 4379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:42.525381 4379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:42.533381 4379 catalog_manager.cc:1347] Generated new cluster ID: 8ccee9f4a9dd4c10a22460a92387ce36
I20260809 15:01:42.533476 4379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:42.553399 4379 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:42.555022 4379 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:42.574733 4379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9e722f98b6473eb98fcabbabf70c26: Generated new TSK 0
I20260809 15:01:42.575647 4379 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 15:01:42.620767 4407 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 15:01:42.621002 4407 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 15:01:42.622094 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:42.600638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.27.186.190:44847)
0809 15:01:42.600817 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:42.600823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:42.601262 (+ 439us) server_negotiation.cc:407] Connection header received
0809 15:01:42.601573 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:42.601578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:42.601679 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:42.601829 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0809 15:01:42.602350 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:42.603471 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:42.604819 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:42.605494 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:42.605624 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:42.606101 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 15:01:42.621477 (+ 15376us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 15:01:42.621800 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 15:01:42.622211 4364 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 15:01:42.623610 4405 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:42.600477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44847 (local address 127.0.0.1:41742)
0809 15:01:42.601172 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:42.601187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:42.601323 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:42.601839 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:42.601843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:42.601853 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 15:01:42.602204 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:42.602215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:42.603588 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:42.603592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:42.604687 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:42.604699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:42.605644 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:42.605649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:42.605768 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:42.605834 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 15:01:42.621724 (+ 15890us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 15:01:42.621966 (+ 242us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.186.190:44847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":391,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.186.190:44847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 15:01:42.625134 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 4259
I20260809 15:01:42.626164 4354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:42.717423 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 4259
2026-08-09T15:01:42Z chronyd exiting
I20260809 15:01:42.774403 28394 test_util.cc:175] -----------------------------------------------
I20260809 15:01:42.774542 28394 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786287660051758-28394-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 15:01:42.778165 28394 mini_oidc.cc:152] Starting JWKS server
I20260809 15:01:42.779889 28394 webserver.cc:460] Webserver started at https://127.0.0.1:40535/ using document root <none> and password file <none>
I20260809 15:01:42.780107 28394 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 15:01:42.780709 28394 webserver.cc:460] Webserver started at http://127.0.0.1:36967/ using document root <none> and password file <none>
2026-08-09T15:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:42Z Disabled control of system clock
I20260809 15:01:42.808439 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33685
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:33685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40535/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/testchainca.cert with env {}
W20260809 15:01:43.124257 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:43.124516 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:43.124657 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:43.131546 4425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:43.131649 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:43.131716 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:43.131793 4425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:43.131851 4425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 15:01:43.131939 4425 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 15:01:43.131990 4425 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40535/jwks.json
I20260809 15:01:43.141852 4425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:33685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40535/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:43.142979 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:43.144186 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:43.154913 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:43.154877 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:43.155064 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:43.156203 4425 server_base.cc:1034] running on GCE node
I20260809 15:01:43.156900 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:43.158557 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:43.159754 4425 hybrid_clock.cc:648] HybridClock initialized: now 1786287703159720 us; error 34 us; skew 500 ppm
I20260809 15:01:43.160058 4425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:43.162248 4425 webserver.cc:460] Webserver started at http://127.27.186.190:46221/ using document root <none> and password file <none>
I20260809 15:01:43.162845 4425 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:43.162950 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:43.163260 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:43.165977 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "c335443dabaa41f8ab08694b69f4a588"
format_stamp: "Formatted at 2026-08-09 15:01:43 on dist-test-slave-q9h9"
server_key: "aa43995fdbaf9667e1f702511a7d41da"
server_key_iv: "c1b6086ce61c2cbaf59cf7492a7056bb"
server_key_version: "encryptionkey@0"
I20260809 15:01:43.166868 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "c335443dabaa41f8ab08694b69f4a588"
format_stamp: "Formatted at 2026-08-09 15:01:43 on dist-test-slave-q9h9"
server_key: "aa43995fdbaf9667e1f702511a7d41da"
server_key_iv: "c1b6086ce61c2cbaf59cf7492a7056bb"
server_key_version: "encryptionkey@0"
I20260809 15:01:43.171923 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 15:01:43.175482 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:43.176699 4425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:43.176843 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "c335443dabaa41f8ab08694b69f4a588"
format_stamp: "Formatted at 2026-08-09 15:01:43 on dist-test-slave-q9h9"
server_key: "aa43995fdbaf9667e1f702511a7d41da"
server_key_iv: "c1b6086ce61c2cbaf59cf7492a7056bb"
server_key_version: "encryptionkey@0"
I20260809 15:01:43.177034 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:43.199128 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:43.199810 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:43.200067 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:43.216375 4425 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:33685
I20260809 15:01:43.216414 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:33685 every 8 connection(s)
I20260809 15:01:43.217530 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:43.220832 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:43.226289 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 4425
I20260809 15:01:43.226455 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:43.227056 28394 external_mini_cluster.cc:1428] Setting key 8069b375f185bc4dcbdd287b30576bf0
I20260809 15:01:43.231472 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588: Bootstrap starting.
I20260809 15:01:43.235503 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:43.236593 4523 log.cc:824] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:43.239485 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588: No bootstrap required, opened a new log
I20260809 15:01:43.241905 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:43.228896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.27.186.190:33685)
0809 15:01:43.229884 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:43.229897 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:43.229950 (+ 53us) server_negotiation.cc:407] Connection header received
0809 15:01:43.232360 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:43.232388 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:43.232946 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:43.233403 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:43.233927 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.235950 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.237307 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.238099 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.238264 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:43.239277 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:43.239288 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:43.239304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:43.239986 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:43.240090 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:43.240115 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:43.240227 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:43.240428 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 15:01:43.240711 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":281,"threads_started":1}
I20260809 15:01:43.245340 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:01:43.247560 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c335443dabaa41f8ab08694b69f4a588" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33685 } }
I20260809 15:01:43.247889 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:43.247982 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c335443dabaa41f8ab08694b69f4a588, State: Initialized, Role: FOLLOWER
I20260809 15:01:43.248561 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c335443dabaa41f8ab08694b69f4a588" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33685 } }
I20260809 15:01:43.248776 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:43.248906 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:43.249051 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:43.251148 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c335443dabaa41f8ab08694b69f4a588" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33685 } }
I20260809 15:01:43.251578 4523 leader_election.cc:302] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c335443dabaa41f8ab08694b69f4a588; no voters:
I20260809 15:01:43.252162 4523 leader_election.cc:288] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:43.252348 4530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:43.253057 4530 raft_consensus.cc:686] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [term 1 LEADER]: Becoming Leader. State: Replica: c335443dabaa41f8ab08694b69f4a588, State: Running, Role: LEADER
I20260809 15:01:43.253530 4530 consensus_queue.cc:237] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c335443dabaa41f8ab08694b69f4a588" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33685 } }
I20260809 15:01:43.254172 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:43.256165 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c335443dabaa41f8ab08694b69f4a588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c335443dabaa41f8ab08694b69f4a588" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33685 } } }
I20260809 15:01:43.256528 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:43.256947 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c335443dabaa41f8ab08694b69f4a588. Latest consensus state: current_term: 1 leader_uuid: "c335443dabaa41f8ab08694b69f4a588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c335443dabaa41f8ab08694b69f4a588" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33685 } } }
I20260809 15:01:43.257241 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:43.262264 4539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:43.267127 4539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:43.275566 4539 catalog_manager.cc:1347] Generated new cluster ID: b06b2ad003524791a64cd6b474c100ef
I20260809 15:01:43.275653 4539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:43.292769 4539 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:43.293900 4539 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:43.301395 4539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c335443dabaa41f8ab08694b69f4a588: Generated new TSK 0
I20260809 15:01:43.302099 4539 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 15:01:43.338064 4565 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 15:01:43.338181 4565 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 15:01:43.339279 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:43.320311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.27.186.190:33685)
0809 15:01:43.320485 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:43.320492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:43.320865 (+ 373us) server_negotiation.cc:407] Connection header received
0809 15:01:43.321057 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:43.321062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:43.321152 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:43.321308 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0809 15:01:43.321700 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.322846 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.324110 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.324780 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.324900 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:43.325300 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 15:01:43.338678 (+ 13378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 15:01:43.339017 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 15:01:43.339301 4562 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:43.320162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33685 (local address 127.0.0.1:38062)
0809 15:01:43.320775 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:43.320793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:43.320904 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:43.321308 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:43.321311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:43.321320 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 15:01:43.321561 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:43.321569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.322962 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.322966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:43.323980 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:43.323991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.324913 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.324917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:43.325043 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:43.325072 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 15:01:43.338860 (+ 13788us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 15:01:43.339026 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.186.190:33685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":353,"threads_started":1}
W20260809 15:01:43.339395 4528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260809 15:01:43.339921 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 4425
I20260809 15:01:43.340909 4517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:43.430750 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 4425
2026-08-09T15:01:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 15:01:43.487797 28394 mini_oidc.cc:152] Starting JWKS server
I20260809 15:01:43.489585 28394 webserver.cc:460] Webserver started at https://127.0.0.1:39493/ using document root <none> and password file <none>
I20260809 15:01:43.489817 28394 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 15:01:43.490523 28394 webserver.cc:460] Webserver started at http://127.0.0.1:41289/ using document root <none> and password file <none>
2026-08-09T15:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:43Z Disabled control of system clock
I20260809 15:01:43.518615 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:39239
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:39239
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39493/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 15:01:43.819892 4576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:43.820163 4576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:43.820294 4576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:43.827054 4576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:43.827157 4576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:43.827222 4576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:43.827301 4576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:43.827404 4576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 15:01:43.827484 4576 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 15:01:43.827540 4576 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39493/jwks.json
I20260809 15:01:43.835978 4576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:39239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:39239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39493/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:43.837035 4576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:43.838601 4576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:43.849046 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:43.849081 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:43.849004 4586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:43.849946 4576 server_base.cc:1034] running on GCE node
I20260809 15:01:43.850662 4576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:43.851908 4576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:43.853106 4576 hybrid_clock.cc:648] HybridClock initialized: now 1786287703853068 us; error 30 us; skew 500 ppm
I20260809 15:01:43.853418 4576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:43.855504 4576 webserver.cc:460] Webserver started at http://127.27.186.190:42907/ using document root <none> and password file <none>
I20260809 15:01:43.856102 4576 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:43.856196 4576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:43.856504 4576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:43.859126 4576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4"
format_stamp: "Formatted at 2026-08-09 15:01:43 on dist-test-slave-q9h9"
server_key: "d66e0dea250a791a81388364ea825baa"
server_key_iv: "3ef096c52852ad142ea2273f7262d73e"
server_key_version: "encryptionkey@0"
I20260809 15:01:43.859850 4576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4"
format_stamp: "Formatted at 2026-08-09 15:01:43 on dist-test-slave-q9h9"
server_key: "d66e0dea250a791a81388364ea825baa"
server_key_iv: "3ef096c52852ad142ea2273f7262d73e"
server_key_version: "encryptionkey@0"
I20260809 15:01:43.864446 4576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 15:01:43.867534 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:43.868690 4576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 15:01:43.868849 4576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4"
format_stamp: "Formatted at 2026-08-09 15:01:43 on dist-test-slave-q9h9"
server_key: "d66e0dea250a791a81388364ea825baa"
server_key_iv: "3ef096c52852ad142ea2273f7262d73e"
server_key_version: "encryptionkey@0"
I20260809 15:01:43.869035 4576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:43.879864 4576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:43.880530 4576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:43.880787 4576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:43.895886 4576 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:39239
I20260809 15:01:43.895905 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:39239 every 8 connection(s)
I20260809 15:01:43.896961 4576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:43.900296 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:43.902186 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 4576
I20260809 15:01:43.902369 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:43.902956 28394 external_mini_cluster.cc:1428] Setting key fc4427c00f205330ab12a94ec0a87180
I20260809 15:01:43.912442 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4: Bootstrap starting.
I20260809 15:01:43.915942 4693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:43.917145 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:43.904593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.27.186.190:39239)
0809 15:01:43.905623 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:43.905636 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:43.905688 (+ 52us) server_negotiation.cc:407] Connection header received
0809 15:01:43.907882 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:43.907908 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:43.908448 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:43.908856 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:43.909364 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.911342 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.912555 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.913501 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.913668 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:43.914631 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:43.914640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:43.914655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:43.915296 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:43.915408 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:43.915433 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:43.915543 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:43.915669 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 15:01:43.915961 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":324,"threads_started":1}
I20260809 15:01:43.917172 4693 log.cc:824] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:43.920172 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4: No bootstrap required, opened a new log
I20260809 15:01:43.921053 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:01:43.928160 4693 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 39239 } }
I20260809 15:01:43.928447 4693 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:43.928516 4693 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc87c7a89c74b0097fca5ff2b1a46e4, State: Initialized, Role: FOLLOWER
I20260809 15:01:43.928992 4693 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 39239 } }
I20260809 15:01:43.929188 4693 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:43.929308 4693 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:43.929459 4693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:43.931514 4693 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 39239 } }
I20260809 15:01:43.931943 4693 leader_election.cc:302] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc87c7a89c74b0097fca5ff2b1a46e4; no voters:
I20260809 15:01:43.932464 4693 leader_election.cc:288] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:43.932567 4705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:43.933166 4705 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [term 1 LEADER]: Becoming Leader. State: Replica: cbc87c7a89c74b0097fca5ff2b1a46e4, State: Running, Role: LEADER
I20260809 15:01:43.933640 4705 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 39239 } }
I20260809 15:01:43.934276 4693 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:43.936354 4712 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc87c7a89c74b0097fca5ff2b1a46e4. Latest consensus state: current_term: 1 leader_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 39239 } } }
I20260809 15:01:43.936329 4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc87c7a89c74b0097fca5ff2b1a46e4" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 39239 } } }
I20260809 15:01:43.936762 4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:43.937381 4712 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:43.938612 4718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:43.943532 4718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:43.950946 4718 catalog_manager.cc:1347] Generated new cluster ID: 1028845b5cc747f39f1911e8678ebc8a
I20260809 15:01:43.951033 4718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:43.968325 4718 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:43.969376 4718 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:43.976596 4718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc87c7a89c74b0097fca5ff2b1a46e4: Generated new TSK 0
I20260809 15:01:43.977182 4718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:01:44.002380 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:43.994601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.27.186.190:39239)
0809 15:01:43.994769 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:43.994775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:43.995152 (+ 377us) server_negotiation.cc:407] Connection header received
0809 15:01:43.995333 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:43.995338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:43.995418 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:43.995557 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0809 15:01:43.995976 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.997056 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.998409 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.999085 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.999215 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:43.999681 (+ 466us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 15:01:44.001769 (+ 2088us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 15:01:44.002087 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 15:01:44.002496 4702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 15:01:44.004038 4745 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:43.994476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:39239 (local address 127.0.0.1:53572)
0809 15:01:43.995063 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:43.995077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:43.995185 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:43.995556 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:43.995559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:43.995567 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 15:01:43.995839 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:43.995847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.997184 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.997188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:43.998283 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:43.998295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:43.999249 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:43.999254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:43.999391 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:43.999432 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 15:01:44.001955 (+ 2523us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 15:01:44.003640 (+ 1685us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.186.190:39239: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":1397,"thread_start_us":338,"threads_started":1}
I20260809 15:01:44.004904 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 4576
I20260809 15:01:44.005919 4686 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 15:01:44.013926 4747 webserver.cc:533] Webserver: sslize failed:
E20260809 15:01:44.014029 4747 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 15:01:44.092502 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 4576
2026-08-09T15:01:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (665 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T15:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:44Z Disabled control of system clock
I20260809 15:01:44.179795 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:41213
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:40583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:41213
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/testchainca.cert with env {}
W20260809 15:01:44.484048 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:44.484340 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:44.484470 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:44.491324 4756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:44.491415 4756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:44.491477 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:44.491554 4756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:44.491611 4756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 15:01:44.491690 4756 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 15:01:44.491739 4756 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 15:01:44.500974 4756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:41213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:44.502060 4756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:44.503187 4756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:44.513764 4766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:44.513852 4769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:44.513875 4767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:44.514417 4756 server_base.cc:1034] running on GCE node
I20260809 15:01:44.515080 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:44.516244 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:44.517436 4756 hybrid_clock.cc:648] HybridClock initialized: now 1786287704517398 us; error 29 us; skew 500 ppm
I20260809 15:01:44.517740 4756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:01:44.519752 4756 webserver.cc:460] Webserver started at http://127.27.186.190:39805/ using document root <none> and password file <none>
I20260809 15:01:44.520365 4756 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:44.520463 4756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:44.520730 4756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:44.523324 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "a60706c761ee4963b513fd760ba3568f"
format_stamp: "Formatted at 2026-08-09 15:01:44 on dist-test-slave-q9h9"
server_key: "ea80e7b7e4b8482f71c8d38941918eed"
server_key_iv: "c63f09f850f242da6f938981a3473216"
server_key_version: "encryptionkey@0"
I20260809 15:01:44.524044 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "a60706c761ee4963b513fd760ba3568f"
format_stamp: "Formatted at 2026-08-09 15:01:44 on dist-test-slave-q9h9"
server_key: "ea80e7b7e4b8482f71c8d38941918eed"
server_key_iv: "c63f09f850f242da6f938981a3473216"
server_key_version: "encryptionkey@0"
I20260809 15:01:44.528578 4756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 15:01:44.531646 4780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:44.532793 4756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:44.532941 4756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "a60706c761ee4963b513fd760ba3568f"
format_stamp: "Formatted at 2026-08-09 15:01:44 on dist-test-slave-q9h9"
server_key: "ea80e7b7e4b8482f71c8d38941918eed"
server_key_iv: "c63f09f850f242da6f938981a3473216"
server_key_version: "encryptionkey@0"
I20260809 15:01:44.533121 4756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:44.558916 4756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:44.559571 4756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:44.559801 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:44.575954 4756 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:41213
I20260809 15:01:44.575973 4852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:41213 every 8 connection(s)
I20260809 15:01:44.577049 4756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:44.580272 4853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:44.585304 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 4756
I20260809 15:01:44.585469 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:44.586110 28394 external_mini_cluster.cc:1428] Setting key c0aacd9dce9262055be2f9a36bbba4c7
I20260809 15:01:44.589799 4853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f: Bootstrap starting.
I20260809 15:01:44.592784 4853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:44.593657 4853 log.cc:824] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:44.596190 4853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f: No bootstrap required, opened a new log
I20260809 15:01:44.598093 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:44.587841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.27.186.190:41213)
0809 15:01:44.588731 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:44.588741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:44.588777 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:01:44.590394 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:44.590415 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:44.590787 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:44.591094 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:44.591672 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:44.592873 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:44.594113 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:44.594766 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:44.594962 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:44.596021 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:44.596028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:44.596040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:01:44.596531 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:44.596608 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:01:44.596627 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:44.596723 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:44.596945 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 15:01:44.597201 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":345,"threads_started":1}
I20260809 15:01:44.601573 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:01:44.603929 4853 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60706c761ee4963b513fd760ba3568f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41213 } }
I20260809 15:01:44.604245 4853 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:44.604347 4853 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60706c761ee4963b513fd760ba3568f, State: Initialized, Role: FOLLOWER
I20260809 15:01:44.604874 4853 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60706c761ee4963b513fd760ba3568f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41213 } }
I20260809 15:01:44.605067 4853 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:44.605156 4853 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:44.605288 4853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:44.607353 4853 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60706c761ee4963b513fd760ba3568f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41213 } }
I20260809 15:01:44.607838 4853 leader_election.cc:302] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60706c761ee4963b513fd760ba3568f; no voters:
I20260809 15:01:44.608309 4853 leader_election.cc:288] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:44.608429 4866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:44.609067 4866 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [term 1 LEADER]: Becoming Leader. State: Replica: a60706c761ee4963b513fd760ba3568f, State: Running, Role: LEADER
I20260809 15:01:44.609542 4866 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60706c761ee4963b513fd760ba3568f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41213 } }
I20260809 15:01:44.610005 4853 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:44.611836 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60706c761ee4963b513fd760ba3568f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60706c761ee4963b513fd760ba3568f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41213 } } }
I20260809 15:01:44.612180 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:44.612655 4868 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60706c761ee4963b513fd760ba3568f. Latest consensus state: current_term: 1 leader_uuid: "a60706c761ee4963b513fd760ba3568f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60706c761ee4963b513fd760ba3568f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41213 } } }
I20260809 15:01:44.612975 4868 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:44.614328 4877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:44.619268 4877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:44.626891 4877 catalog_manager.cc:1347] Generated new cluster ID: 981589a078a04bbf85a1fb38f312b2d8
I20260809 15:01:44.626991 4877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:44.641906 4877 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:44.643462 4877 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:44.666064 4877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60706c761ee4963b513fd760ba3568f: Generated new TSK 0
I20260809 15:01:44.667176 4877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:01:45.144613 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:44.684863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.27.186.190:41213)
0809 15:01:44.685031 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:44.685037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:44.685346 (+ 309us) server_negotiation.cc:407] Connection header received
0809 15:01:44.685518 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:44.685524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:44.685628 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:44.685781 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0809 15:01:44.686177 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:44.687279 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:44.688581 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:44.689260 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:44.689381 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:44.689790 (+ 409us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 15:01:45.144048 (+454258us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 15:01:45.144376 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52048: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 15:01:45.144733 4861 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52048: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 15:01:45.144944 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:44.684744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41213 (local address 127.0.0.1:52048)
0809 15:01:44.685256 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:44.685270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:44.685379 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:44.685769 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:44.685771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:44.685780 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 15:01:44.686043 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:44.686059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:44.687412 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:44.687416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:44.688450 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:44.688461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:44.689393 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:44.689396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:44.689510 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:44.689540 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 15:01:45.144324 (+454784us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 15:01:45.144563 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.186.190:41213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
I20260809 15:01:45.146085 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 4756
I20260809 15:01:45.147241 4846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:45.248404 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 4756
2026-08-09T15:01:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1139 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (389 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (391 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:46 dist-test-slave-q9h9 krb5kdc[4941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:46 dist-test-slave-q9h9 krb5kdc[4941](info): set up 2 sockets
Aug 09 15:01:46 dist-test-slave-q9h9 krb5kdc[4941](info): commencing operation
krb5kdc: starting...
W20260809 15:01:47.240597 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:47 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:01:47.429497 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37037
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:37037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:47.738077 4959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:47.738313 4959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:47.738402 4959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:47.745158 4959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:47.745244 4959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:47.745294 4959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:47.745347 4959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:47.745389 4959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:47.754405 4959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:37037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:47.755416 4959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:47.756528 4959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:47.767028 4964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:47.767074 4971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:47.767094 4968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:47.767752 4959 server_base.cc:1034] running on GCE node
I20260809 15:01:47.768467 4959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:47.769656 4959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:47.770851 4959 hybrid_clock.cc:648] HybridClock initialized: now 1786287707770804 us; error 31 us; skew 500 ppm
I20260809 15:01:47.771154 4959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:47 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287707, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:47.774001 4959 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:01:47.775421 4959 webserver.cc:460] Webserver started at http://127.27.186.190:39249/ using document root <none> and password file <none>
I20260809 15:01:47.776029 4959 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:47.776142 4959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:47.776432 4959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:47.779160 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "b322364bbb4a4479b256b172a42788e8"
format_stamp: "Formatted at 2026-08-09 15:01:47 on dist-test-slave-q9h9"
server_key: "4df1442d477b2ecc300a12ccee035e53"
server_key_iv: "30adeb26a0d7d35a7be3a1f0386dff4d"
server_key_version: "encryptionkey@0"
I20260809 15:01:47.779868 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "b322364bbb4a4479b256b172a42788e8"
format_stamp: "Formatted at 2026-08-09 15:01:47 on dist-test-slave-q9h9"
server_key: "4df1442d477b2ecc300a12ccee035e53"
server_key_iv: "30adeb26a0d7d35a7be3a1f0386dff4d"
server_key_version: "encryptionkey@0"
I20260809 15:01:47.784438 4959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 15:01:47.787561 4984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:47.788664 4959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:47.788820 4959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "b322364bbb4a4479b256b172a42788e8"
format_stamp: "Formatted at 2026-08-09 15:01:47 on dist-test-slave-q9h9"
server_key: "4df1442d477b2ecc300a12ccee035e53"
server_key_iv: "30adeb26a0d7d35a7be3a1f0386dff4d"
server_key_version: "encryptionkey@0"
I20260809 15:01:47.788985 4959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:47.799440 4959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:47.803166 4959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:47.803412 4959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:47.819878 4959 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:37037
I20260809 15:01:47.819914 5062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:37037 every 8 connection(s)
I20260809 15:01:47.821053 4959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:47.824370 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:47.825062 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 4959
I20260809 15:01:47.825311 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:47.826174 28394 external_mini_cluster.cc:1428] Setting key 67db6e076d5104e61a2038e6c4297479
I20260809 15:01:47.838259 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8: Bootstrap starting.
Aug 09 15:01:47 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:47.841871 5064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:47.842911 5064 log.cc:824] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:47.845845 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8: No bootstrap required, opened a new log
I20260809 15:01:47.848568 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:47.828176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.27.186.190:37037)
0809 15:01:47.828937 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:47.828950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:47.828998 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:01:47.830973 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:47.830994 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:47.831350 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:47.831650 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:47.833299 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:47.834588 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:47.836325 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:47.836926 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:47.837048 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:47.841577 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:47.841584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:47.841597 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:47.841637 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:47.844760 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:47.845514 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:47.845520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:47.845525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:47.845675 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:47.846112 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:47.846116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:47.846118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:47.846606 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:47.846821 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:47.847256 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 15:01:47.847563 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260809 15:01:47.853381 5064 raft_consensus.cc:348] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b322364bbb4a4479b256b172a42788e8" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37037 } }
I20260809 15:01:47.853724 5064 raft_consensus.cc:374] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:47.853811 5064 raft_consensus.cc:729] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b322364bbb4a4479b256b172a42788e8, State: Initialized, Role: FOLLOWER
I20260809 15:01:47.854362 5064 consensus_queue.cc:260] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b322364bbb4a4479b256b172a42788e8" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37037 } }
I20260809 15:01:47.854591 5064 raft_consensus.cc:388] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:47.854724 5064 raft_consensus.cc:482] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:47.854880 5064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:47.856900 5064 raft_consensus.cc:504] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b322364bbb4a4479b256b172a42788e8" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37037 } }
I20260809 15:01:47.857345 5064 leader_election.cc:302] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b322364bbb4a4479b256b172a42788e8; no voters:
I20260809 15:01:47.857875 5064 leader_election.cc:288] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:47.857980 5071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:47.858631 5071 raft_consensus.cc:686] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [term 1 LEADER]: Becoming Leader. State: Replica: b322364bbb4a4479b256b172a42788e8, State: Running, Role: LEADER
I20260809 15:01:47.859169 5071 consensus_queue.cc:237] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b322364bbb4a4479b256b172a42788e8" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37037 } }
I20260809 15:01:47.859795 5064 sys_catalog.cc:564] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:47.861939 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b322364bbb4a4479b256b172a42788e8. Latest consensus state: current_term: 1 leader_uuid: "b322364bbb4a4479b256b172a42788e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b322364bbb4a4479b256b172a42788e8" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37037 } } }
I20260809 15:01:47.862396 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:47.861963 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b322364bbb4a4479b256b172a42788e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b322364bbb4a4479b256b172a42788e8" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37037 } } }
I20260809 15:01:47.862578 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:47.863933 5083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:47.868261 5083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:47.876374 5083 catalog_manager.cc:1347] Generated new cluster ID: e9f2911e86b549d2b6b312b1e411d886
I20260809 15:01:47.876461 5083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:47.889266 5083 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:47.890446 5083 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:47.899588 5083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b322364bbb4a4479b256b172a42788e8: Generated new TSK 0
I20260809 15:01:47.900250 5083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:01:47.950714 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37037
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:48.270161 5108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:48.270423 5108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:48.270565 5108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:48.277558 5108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:48.277661 5108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:48.277810 5108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:48.287132 5108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:48.288314 5108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:48.289569 5108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:48.301963 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:48.302372 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:48.301991 5123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:48.302860 5108 server_base.cc:1034] running on GCE node
I20260809 15:01:48.303406 5108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:48.304276 5108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:48.305466 5108 hybrid_clock.cc:648] HybridClock initialized: now 1786287708305421 us; error 36 us; skew 500 ppm
I20260809 15:01:48.305760 5108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:48 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:48.311693 5108 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:01:48.313022 5108 webserver.cc:460] Webserver started at http://127.27.186.129:36629/ using document root <none> and password file <none>
I20260809 15:01:48.313638 5108 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:48.313766 5108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:48.314090 5108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:48.316869 5108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "34d3ba951f354578a6d2990db6e8b7c7"
format_stamp: "Formatted at 2026-08-09 15:01:48 on dist-test-slave-q9h9"
server_key: "3f5bc8cae328776467664de16c293f7b"
server_key_iv: "bc2d173b6059ac0162af21cd8500d079"
server_key_version: "encryptionkey@0"
I20260809 15:01:48.317669 5108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "34d3ba951f354578a6d2990db6e8b7c7"
format_stamp: "Formatted at 2026-08-09 15:01:48 on dist-test-slave-q9h9"
server_key: "3f5bc8cae328776467664de16c293f7b"
server_key_iv: "bc2d173b6059ac0162af21cd8500d079"
server_key_version: "encryptionkey@0"
I20260809 15:01:48.322561 5108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 15:01:48.325793 5135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:48.326992 5108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:48.327135 5108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "34d3ba951f354578a6d2990db6e8b7c7"
format_stamp: "Formatted at 2026-08-09 15:01:48 on dist-test-slave-q9h9"
server_key: "3f5bc8cae328776467664de16c293f7b"
server_key_iv: "bc2d173b6059ac0162af21cd8500d079"
server_key_version: "encryptionkey@0"
I20260809 15:01:48.327310 5108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:48.341578 5108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:48.345104 5108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:48.345356 5108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:48.346254 5108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:48.347502 5108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:48.347589 5108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:48.348641 5108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:48.348722 5108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:48.378700 5108 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:46463
I20260809 15:01:48.378741 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:46463 every 8 connection(s)
I20260809 15:01:48.379870 5108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:48.388569 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 5108
I20260809 15:01:48.388751 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:48.389382 28394 external_mini_cluster.cc:1428] Setting key 1571e2e0c9025d4e4d4c67cb46031551
Aug 09 15:01:48 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:48.399761 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:48.382219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:46271 (local address 127.27.186.190:37037)
0809 15:01:48.382649 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:48.382654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:48.383487 (+ 833us) server_negotiation.cc:407] Connection header received
0809 15:01:48.385275 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:48.385281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:48.385360 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:48.385456 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:48.386954 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.387700 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.389253 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.389705 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.389815 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:48.394159 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:48.394165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:48.394170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:48.394219 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:48.396691 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:48.397374 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:48.397378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:48.397380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:48.397499 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:48.397931 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:48.397934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:48.397936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:48.398255 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:48.398391 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:48.399108 (+ 717us) server_negotiation.cc:299] Negotiation successful
0809 15:01:48.399379 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":227,"threads_started":1}
I20260809 15:01:48.400606 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:48.382633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37037 (local address 127.27.186.129:46271)
0809 15:01:48.383334 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:48.383378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:48.385010 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:48.385698 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:48.385706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:48.386242 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:48.386791 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:48.386809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.387846 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.387850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:48.389111 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:48.389124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.389918 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.389926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:48.390220 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:48.390888 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:48.390913 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:48.393907 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:48.396836 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:48.396845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:48.396859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:48.397250 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:48.397603 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:48.397609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:48.397613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:48.397816 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:48.398381 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:48.398388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:48.398715 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 15:01:48.398988 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 15:01:48.399456 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":275,"threads_started":1}
I20260809 15:01:48.402073 5323 heartbeater.cc:347] Connected to a master server at 127.27.186.190:37037
I20260809 15:01:48.402361 5323 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:48.403013 5323 heartbeater.cc:511] Master 127.27.186.190:37037 requested a full tablet report, sending...
I20260809 15:01:48.405146 5013 ts_manager.cc:194] Registered new tserver with Master: 34d3ba951f354578a6d2990db6e8b7c7 (127.27.186.129:46463)
I20260809 15:01:48.406976 5013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:46271
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:01:48.435135 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37037
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:48.744791 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:48.745016 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:48.745141 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:48.751966 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:48.752061 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:48.752216 5331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:48.761309 5331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:48.762319 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:48.763459 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:48.775566 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:48.775640 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:48.776193 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:48.776201 5331 server_base.cc:1034] running on GCE node
I20260809 15:01:48.776844 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:48.777639 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:48.778837 5331 hybrid_clock.cc:648] HybridClock initialized: now 1786287708778797 us; error 33 us; skew 500 ppm
I20260809 15:01:48.779111 5331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:48 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:48.781972 5331 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:01:48.783221 5331 webserver.cc:460] Webserver started at http://127.27.186.130:37085/ using document root <none> and password file <none>
I20260809 15:01:48.783782 5331 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:48.783879 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:48.784127 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:48.786736 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "e5b3c631a5b242418313110f965c4092"
format_stamp: "Formatted at 2026-08-09 15:01:48 on dist-test-slave-q9h9"
server_key: "3010ba8b0061409c8b5c95c746703761"
server_key_iv: "f97a9e1aaa268491a6cc163e365e9629"
server_key_version: "encryptionkey@0"
I20260809 15:01:48.787428 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "e5b3c631a5b242418313110f965c4092"
format_stamp: "Formatted at 2026-08-09 15:01:48 on dist-test-slave-q9h9"
server_key: "3010ba8b0061409c8b5c95c746703761"
server_key_iv: "f97a9e1aaa268491a6cc163e365e9629"
server_key_version: "encryptionkey@0"
I20260809 15:01:48.792094 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:01:48.795514 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:48.796650 5331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 15:01:48.796795 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "e5b3c631a5b242418313110f965c4092"
format_stamp: "Formatted at 2026-08-09 15:01:48 on dist-test-slave-q9h9"
server_key: "3010ba8b0061409c8b5c95c746703761"
server_key_iv: "f97a9e1aaa268491a6cc163e365e9629"
server_key_version: "encryptionkey@0"
I20260809 15:01:48.796963 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:48.816749 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:48.820225 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:48.820479 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:48.821331 5331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:48.822636 5331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:48.822721 5331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:48.822846 5331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:48.822906 5331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:48.853055 5331 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:34449
I20260809 15:01:48.853098 5535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:34449 every 8 connection(s)
I20260809 15:01:48.854229 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:48.860004 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 5331
I20260809 15:01:48.860145 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:48.860606 28394 external_mini_cluster.cc:1428] Setting key 1a3a90a12a4b6ab6a176bfed6c5a1d4b
Aug 09 15:01:48 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:48.873163 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:48.856476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:36783 (local address 127.27.186.190:37037)
0809 15:01:48.856647 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:48.856652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:48.857858 (+ 1206us) server_negotiation.cc:407] Connection header received
0809 15:01:48.859631 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:48.859637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:48.859720 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:48.859807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:48.861306 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.862353 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.863842 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.864249 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.864357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:48.867551 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:48.867558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:48.867563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:48.867610 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:48.870440 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:48.871081 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:48.871087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:48.871091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:48.871257 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:48.871646 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:48.871651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:48.871654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:48.871918 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:48.872054 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:48.872779 (+ 725us) server_negotiation.cc:299] Negotiation successful
0809 15:01:48.872941 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:01:48.874329 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:48.856912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37037 (local address 127.27.186.130:36783)
0809 15:01:48.857691 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:48.857738 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:48.859372 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:48.860037 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:48.860045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:48.860580 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:48.861147 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:48.861165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.862493 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.862497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:48.863707 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:48.863720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:48.864460 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:48.864468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:48.864675 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:48.865256 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:48.865283 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:48.867317 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:48.870560 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:48.870566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:48.870576 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:48.870960 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:48.871362 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:48.871366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:48.871368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:48.871539 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:48.872044 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:48.872051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:48.872410 (+ 359us) client_negotiation.cc:769] Sending connection context
0809 15:01:48.872689 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 15:01:48.873140 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":366,"threads_started":1}
I20260809 15:01:48.875651 5537 heartbeater.cc:347] Connected to a master server at 127.27.186.190:37037
I20260809 15:01:48.875914 5537 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:48.876535 5537 heartbeater.cc:511] Master 127.27.186.190:37037 requested a full tablet report, sending...
I20260809 15:01:48.877835 5013 ts_manager.cc:194] Registered new tserver with Master: e5b3c631a5b242418313110f965c4092 (127.27.186.130:34449)
I20260809 15:01:48.878639 5013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:36783
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:01:48.905792 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37037
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:49.211913 5545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:49.212173 5545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:49.212316 5545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:49.219149 5545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:49.219260 5545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:49.219421 5545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:49.228348 5545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:37037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:49.229468 5545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:49.230742 5545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:49.243079 5556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:49.243177 5554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:49.243192 5553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:49.244096 5545 server_base.cc:1034] running on GCE node
I20260809 15:01:49.244634 5545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:49.245441 5545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:49.246630 5545 hybrid_clock.cc:648] HybridClock initialized: now 1786287709246587 us; error 34 us; skew 500 ppm
I20260809 15:01:49.247140 5545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.250237 5545 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:01:49.251516 5545 webserver.cc:460] Webserver started at http://127.27.186.131:41315/ using document root <none> and password file <none>
I20260809 15:01:49.252122 5545 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:49.252238 5545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:49.252516 5545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:49.255326 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "11a786eb0b2b436284e09237ca1281a9"
format_stamp: "Formatted at 2026-08-09 15:01:49 on dist-test-slave-q9h9"
server_key: "eea079e9eead10c348253ba22c9a72e8"
server_key_iv: "7d8137b06ca3a422d9bfcdc837e6f1d4"
server_key_version: "encryptionkey@0"
I20260809 15:01:49.256047 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "11a786eb0b2b436284e09237ca1281a9"
format_stamp: "Formatted at 2026-08-09 15:01:49 on dist-test-slave-q9h9"
server_key: "eea079e9eead10c348253ba22c9a72e8"
server_key_iv: "7d8137b06ca3a422d9bfcdc837e6f1d4"
server_key_version: "encryptionkey@0"
I20260809 15:01:49.260774 5545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:49.263936 5567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:49.265441 5545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:01:49.265638 5545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "11a786eb0b2b436284e09237ca1281a9"
format_stamp: "Formatted at 2026-08-09 15:01:49 on dist-test-slave-q9h9"
server_key: "eea079e9eead10c348253ba22c9a72e8"
server_key_iv: "7d8137b06ca3a422d9bfcdc837e6f1d4"
server_key_version: "encryptionkey@0"
I20260809 15:01:49.265901 5545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:49.283941 5545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:49.287699 5545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:49.287946 5545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:49.288767 5545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:49.289970 5545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:49.290071 5545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:49.290194 5545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:49.290244 5545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:49.320132 5545 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:35015
I20260809 15:01:49.320252 5744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:35015 every 8 connection(s)
I20260809 15:01:49.321261 5545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:49.330610 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 5545
I20260809 15:01:49.330758 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:49.331241 28394 external_mini_cluster.cc:1428] Setting key c48a53c3c4873ae9620f118806b058c2
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:01:49.339393 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.323502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:58793 (local address 127.27.186.190:37037)
0809 15:01:49.323666 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.323670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.324729 (+ 1059us) server_negotiation.cc:407] Connection header received
0809 15:01:49.326604 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.326610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.326689 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.326775 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.328148 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.328895 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.330527 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.331024 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.331177 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.334694 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.334699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.334703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.334736 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.337024 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.337626 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.337629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.337631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.337739 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.338104 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.338107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.338109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.338375 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.338484 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.338992 (+ 508us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.339159 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:01:49.340191 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.323914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:37037 (local address 127.27.186.131:58793)
0809 15:01:49.324584 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:49.324619 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:49.326322 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:49.327018 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:49.327027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:49.327481 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:49.327982 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:49.327999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.329045 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.329050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:49.330381 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:49.330396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.331245 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.331253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:49.331539 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.332187 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:49.332224 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:49.334513 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:49.337140 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:49.337146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:49.337156 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:49.337537 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:49.337813 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:49.337816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:49.337818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:49.337983 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:49.338466 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:49.338472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:49.338729 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 15:01:49.338931 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 15:01:49.339243 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
I20260809 15:01:49.341337 5745 heartbeater.cc:347] Connected to a master server at 127.27.186.190:37037
I20260809 15:01:49.341578 5745 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:49.342232 5745 heartbeater.cc:511] Master 127.27.186.190:37037 requested a full tablet report, sending...
I20260809 15:01:49.343544 5013 ts_manager.cc:194] Registered new tserver with Master: 11a786eb0b2b436284e09237ca1281a9 (127.27.186.131:35015)
I20260809 15:01:49.344280 5013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:58793
I20260809 15:01:49.347263 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/test-admin.keytab.
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.374522 28394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 15:01:49.380036 5755 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:49.377484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.27.186.129:46463)
0809 15:01:49.378143 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.378151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.378179 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:01:49.378257 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.378266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.378473 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.378609 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.379569 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38210: BlockingRecv error: recv got EOF from 127.0.0.1:38210 (error 108)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
W20260809 15:01:49.380028 28394 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc' of type FILE
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.381870 28394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.383435 28394 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:49.398741 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.385075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.27.186.129:46463)
0809 15:01:49.385256 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.385261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.385486 (+ 225us) server_negotiation.cc:407] Connection header received
0809 15:01:49.385633 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.385638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.385696 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.385769 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.386759 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.387726 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.388908 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.389422 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.389532 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.392833 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.392844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.392855 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.392883 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.396042 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.396867 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.396876 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.396879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.397046 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.397437 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.397440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.397442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.397770 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.397969 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.398263 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.398485 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 15:01:49.403579 5755 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:49.402120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.27.186.129:46463)
0809 15:01:49.402297 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.402302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.402448 (+ 146us) server_negotiation.cc:407] Connection header received
0809 15:01:49.402620 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.402624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.402687 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.402760 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.403374 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38224: BlockingRecv error: recv got EOF from 127.0.0.1:38224 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 15:01:49.404026 28394 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc)
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.405337 28394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.406491 28394 init.cc:295] Successfully reacquired a new kerberos TGT
I20260809 15:01:49.409547 5323 heartbeater.cc:503] Master 127.27.186.190:37037 was elected leader, sending a full tablet report...
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:49.418830 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.407552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.27.186.129:46463)
0809 15:01:49.407702 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.407707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.408008 (+ 301us) server_negotiation.cc:407] Connection header received
0809 15:01:49.408144 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.408148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.408197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.408268 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.409128 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.409800 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.411231 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.411652 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.411755 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.414842 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.414846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.414850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.414876 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.416940 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.417478 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.417481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.417483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.417586 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.417859 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.417861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.417863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.418121 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.418301 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.418503 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.418647 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 15:01:49.422217 5755 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:01:49.420867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.27.186.129:46463)
0809 15:01:49.421019 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.421025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.421249 (+ 224us) server_negotiation.cc:407] Connection header received
0809 15:01:49.421348 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.421351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.421407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.421474 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.422044 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38256: BlockingRecv error: recv got EOF from 127.0.0.1:38256 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 15:01:49.422394 28394 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786287660051758-28394-0/krb5kdc/krb5cc' of type FILE
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.423659 28394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:49.424770 28394 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:01:49.436523 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.425821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.27.186.129:46463)
0809 15:01:49.425988 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.425993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.426235 (+ 242us) server_negotiation.cc:407] Connection header received
0809 15:01:49.426300 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.426304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.426349 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.426422 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.427255 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.427901 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.428980 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.429381 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.429467 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.432507 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.432511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.432514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.432540 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.434654 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.435175 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.435178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.435180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.435289 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.435540 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.435543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.435545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.435767 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.435970 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.436105 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.436249 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:01:49.437644 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 5108
I20260809 15:01:49.448122 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.438217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.27.186.129:46463)
0809 15:01:49.438326 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.438330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.438688 (+ 358us) server_negotiation.cc:407] Connection header received
0809 15:01:49.438811 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.438815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.438865 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.438934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.439783 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.440385 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.441507 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.441891 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.442001 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.443942 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.443946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.443949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.443974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.445914 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.446517 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.446521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.446523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.446656 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.446966 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.446969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.446971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.447188 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.447400 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.447701 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.447877 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 15:01:49.448645 5314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:49.529217 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 5108
I20260809 15:01:49.550938 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 5331
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:01:49.565423 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.551701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.27.186.130:34449)
0809 15:01:49.552188 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.552195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.552220 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:49.552279 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.552285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.552501 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.552629 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.553595 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.554633 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.555849 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.556386 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.556486 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.560084 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.560097 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.560112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.560153 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.562924 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.563630 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.563638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.563640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.563775 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.564101 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.564105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.564107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.564459 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.564690 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.564918 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.565124 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":228,"threads_started":1}
I20260809 15:01:49.566984 5529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:49.648365 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 5331
I20260809 15:01:49.669992 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 5545
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:01:49.684394 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:49.670957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.27.186.131:35015)
0809 15:01:49.671542 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:49.671548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:49.671577 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:01:49.671634 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:49.671640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:49.671841 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:49.671962 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:49.672914 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.673921 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.675150 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:49.675687 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:49.675780 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:49.679256 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:49.679266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:49.679278 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:49.679308 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:49.682237 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.682787 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.682793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.682796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.682907 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:49.683186 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:49.683189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:49.683191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:49.683479 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:49.683709 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:49.683860 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 15:01:49.684079 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
I20260809 15:01:49.685555 5737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:49.764999 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 5545
I20260809 15:01:49.782950 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 4959
I20260809 15:01:49.783972 5057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:49.882894 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 4959
2026-08-09T15:01:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3851 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[5790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[5790](info): set up 2 sockets
Aug 09 15:01:49 dist-test-slave-q9h9 krb5kdc[5790](info): commencing operation
krb5kdc: starting...
W20260809 15:01:51.077500 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:01:51 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:01:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab.
Entry for principal oryx/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab.
I20260809 15:01:51.271054 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:36429
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab
--principal=oryx/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:36429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:51.574419 5808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:51.574651 5808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:51.574740 5808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:51.581476 5808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:51.581568 5808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:51.581620 5808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:51.581674 5808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:51.581717 5808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:51.590729 5808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:36429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab
--principal=oryx/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:36429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:51.591737 5808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:51.592828 5808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:51.603576 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:51.603646 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:51.603622 5817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:51.604468 5808 server_base.cc:1034] running on GCE node
I20260809 15:01:51.605137 5808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:51.606366 5808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:51.607564 5808 hybrid_clock.cc:648] HybridClock initialized: now 1786287711607523 us; error 24 us; skew 500 ppm
I20260809 15:01:51.607867 5808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:51 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:51.610661 5808 init.cc:377] Logged in from keytab as oryx/127.27.186.190@KRBTEST.COM (short username oryx)
I20260809 15:01:51.612015 5808 webserver.cc:460] Webserver started at http://127.27.186.190:43547/ using document root <none> and password file <none>
I20260809 15:01:51.612601 5808 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:51.612699 5808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:51.612929 5808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:51.615620 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "8df908b68a034b08b5862e491b99c8a9"
format_stamp: "Formatted at 2026-08-09 15:01:51 on dist-test-slave-q9h9"
server_key: "4055c936392a878d48cfad2f37b353ee"
server_key_iv: "0d20fa053d3eacbefe9842bff19b6287"
server_key_version: "encryptionkey@0"
I20260809 15:01:51.616286 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "8df908b68a034b08b5862e491b99c8a9"
format_stamp: "Formatted at 2026-08-09 15:01:51 on dist-test-slave-q9h9"
server_key: "4055c936392a878d48cfad2f37b353ee"
server_key_iv: "0d20fa053d3eacbefe9842bff19b6287"
server_key_version: "encryptionkey@0"
I20260809 15:01:51.620838 5808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 15:01:51.623875 5830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:51.624996 5808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:51.625134 5808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "8df908b68a034b08b5862e491b99c8a9"
format_stamp: "Formatted at 2026-08-09 15:01:51 on dist-test-slave-q9h9"
server_key: "4055c936392a878d48cfad2f37b353ee"
server_key_iv: "0d20fa053d3eacbefe9842bff19b6287"
server_key_version: "encryptionkey@0"
I20260809 15:01:51.625308 5808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:51.648711 5808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:51.652124 5808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:51.652376 5808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:51.668912 5808 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:36429
I20260809 15:01:51.668946 5915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:36429 every 8 connection(s)
I20260809 15:01:51.670118 5808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:01:51.673367 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:51.676672 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 5808
I20260809 15:01:51.676880 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:01:51.677686 28394 external_mini_cluster.cc:1428] Setting key 6a7fe31c1300ada762e587051d9979c4
I20260809 15:01:51.685606 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Bootstrap starting.
I20260809 15:01:51.689054 5916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:51.690289 5916 log.cc:824] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:01:51 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:01:51.692996 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: No bootstrap required, opened a new log
I20260809 15:01:51.700296 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:51.679617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.27.186.190:36429)
0809 15:01:51.680309 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:51.680320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:51.680357 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:01:51.681865 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:51.681886 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:51.682286 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:51.682591 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:51.684348 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:51.685545 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:51.687297 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:51.687912 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:51.688036 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:51.692334 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:51.692346 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:51.692362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:51.692413 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:51.696182 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:51.696819 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:51.696827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:51.696834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:51.697038 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:51.697411 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:51.697417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:51.697420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:51.698092 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:51.698412 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:51.698680 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 15:01:51.699084 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":328,"threads_started":1}
I20260809 15:01:51.702991 5916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } }
I20260809 15:01:51.703306 5916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:51.703379 5916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df908b68a034b08b5862e491b99c8a9, State: Initialized, Role: FOLLOWER
I20260809 15:01:51.703817 5916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } }
I20260809 15:01:51.704008 5916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:01:51.704097 5916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:01:51.704211 5916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:51.706115 5916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } }
I20260809 15:01:51.706528 5916 leader_election.cc:302] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df908b68a034b08b5862e491b99c8a9; no voters:
I20260809 15:01:51.707003 5916 leader_election.cc:288] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:01:51.707111 5925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:51.707755 5925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 LEADER]: Becoming Leader. State: Replica: 8df908b68a034b08b5862e491b99c8a9, State: Running, Role: LEADER
I20260809 15:01:51.708271 5925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } }
I20260809 15:01:51.708710 5916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:51.711045 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } }
I20260809 15:01:51.711606 5929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df908b68a034b08b5862e491b99c8a9. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } }
I20260809 15:01:51.711927 5929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:51.712318 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:51.714373 5939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:51.719391 5939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:51.727208 5939 catalog_manager.cc:1347] Generated new cluster ID: 8450f063ff444687a284745672ea6993
I20260809 15:01:51.727310 5939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:51.751911 5939 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:01:51.752976 5939 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:51.760674 5939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Generated new TSK 0
I20260809 15:01:51.761282 5939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab.
Entry for principal oryx/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab.
I20260809 15:01:51.813527 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36429
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:52.127892 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:52.128218 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:52.128367 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:52.135545 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:52.135651 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:52.135774 5960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:52.145219 5960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:52.146363 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:52.147593 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:52.160224 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:52.160251 5967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:52.160302 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:52.160813 5960 server_base.cc:1034] running on GCE node
I20260809 15:01:52.161444 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:52.162351 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:52.163544 5960 hybrid_clock.cc:648] HybridClock initialized: now 1786287712163497 us; error 41 us; skew 500 ppm
I20260809 15:01:52.163841 5960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:52 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287712, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:52.166759 5960 init.cc:377] Logged in from keytab as oryx/127.27.186.129@KRBTEST.COM (short username oryx)
I20260809 15:01:52.168064 5960 webserver.cc:460] Webserver started at http://127.27.186.129:43163/ using document root <none> and password file <none>
I20260809 15:01:52.168658 5960 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:52.168785 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:52.169063 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:52.171850 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "cc9d3ce19fa54354a136d05df2e352d0"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "2eba85a0eb50505429549adfee7a660c"
server_key_iv: "be02c50d309d4ed0aaeb7a91eba0f88c"
server_key_version: "encryptionkey@0"
I20260809 15:01:52.172582 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9d3ce19fa54354a136d05df2e352d0"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "2eba85a0eb50505429549adfee7a660c"
server_key_iv: "be02c50d309d4ed0aaeb7a91eba0f88c"
server_key_version: "encryptionkey@0"
I20260809 15:01:52.177359 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:01:52.180579 5983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:52.181747 5960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:52.181891 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "cc9d3ce19fa54354a136d05df2e352d0"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "2eba85a0eb50505429549adfee7a660c"
server_key_iv: "be02c50d309d4ed0aaeb7a91eba0f88c"
server_key_version: "encryptionkey@0"
I20260809 15:01:52.182094 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:52.211809 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:52.215451 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:52.215684 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:52.216475 5960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:52.217626 5960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:52.217705 5960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:52.217790 5960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:52.217849 5960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:52.247905 5960 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:36657
I20260809 15:01:52.247946 6178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:36657 every 8 connection(s)
I20260809 15:01:52.249094 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:52.250581 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 5960
I20260809 15:01:52.250824 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:01:52.251549 28394 external_mini_cluster.cc:1428] Setting key 0490af8ac17a7a7e037eb0f5c4504c26
Aug 09 15:01:52 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287712, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:01:52.268157 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:52.251658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:56175 (local address 127.27.186.190:36429)
0809 15:01:52.252150 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:52.252156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:52.253292 (+ 1136us) server_negotiation.cc:407] Connection header received
0809 15:01:52.255180 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:52.255185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:52.255281 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:52.255377 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:52.256822 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.257577 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.259132 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.259525 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.259635 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:52.262927 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:52.262931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:52.262935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:52.262967 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:52.265312 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:52.265981 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:52.265985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:52.265987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:52.266145 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:52.266553 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:52.266557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:52.266559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:52.266819 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:52.266950 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:52.267593 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 15:01:52.267807 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260809 15:01:52.269043 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:52.252174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:56175)
0809 15:01:52.252999 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:52.253042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:52.254746 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:52.255627 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:52.255636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:52.256127 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:52.256652 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:52.256670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.257735 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.257739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:52.258993 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:52.259006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.259740 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.259748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:52.259987 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:52.260600 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:52.260622 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:52.262710 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:52.265440 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:52.265448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:52.265462 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:52.265861 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:52.266247 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:52.266252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:52.266257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:52.266439 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:52.266944 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:52.266952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:52.267237 (+ 285us) client_negotiation.cc:769] Sending connection context
0809 15:01:52.267465 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 15:01:52.267954 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
I20260809 15:01:52.270455 6180 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36429
I20260809 15:01:52.270784 6180 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:52.271646 6180 heartbeater.cc:511] Master 127.27.186.190:36429 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.27.186.130@KRBTEST.COM; defaulting to no policy
I20260809 15:01:52.273972 5855 ts_manager.cc:194] Registered new tserver with Master: cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657)
I20260809 15:01:52.275424 5855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:56175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab.
Entry for principal oryx/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab.
I20260809 15:01:52.298568 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36429
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:52.609544 6192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:52.609803 6192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:52.609905 6192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:52.617097 6192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:52.617193 6192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:52.617314 6192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:52.626719 6192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:52.627803 6192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:52.628997 6192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:52.641144 6197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:52.641191 6200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:52.641233 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:52.642496 6192 server_base.cc:1034] running on GCE node
I20260809 15:01:52.643035 6192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:52.643877 6192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:52.645063 6192 hybrid_clock.cc:648] HybridClock initialized: now 1786287712645016 us; error 40 us; skew 500 ppm
I20260809 15:01:52.645349 6192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:52 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287712, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:52.648299 6192 init.cc:377] Logged in from keytab as oryx/127.27.186.130@KRBTEST.COM (short username oryx)
I20260809 15:01:52.649590 6192 webserver.cc:460] Webserver started at http://127.27.186.130:32935/ using document root <none> and password file <none>
I20260809 15:01:52.650252 6192 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:52.650360 6192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:52.650599 6192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:52.653227 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "065727ab9d7f4b61bda3adb6a82a66ac"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "931ffd99a6d180e27a02a2b8c6980911"
server_key_iv: "9366a029e1e60e1562db7e9e8c117519"
server_key_version: "encryptionkey@0"
I20260809 15:01:52.653929 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "065727ab9d7f4b61bda3adb6a82a66ac"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "931ffd99a6d180e27a02a2b8c6980911"
server_key_iv: "9366a029e1e60e1562db7e9e8c117519"
server_key_version: "encryptionkey@0"
I20260809 15:01:52.658710 6192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:01:52.661953 6209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:52.663156 6192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:01:52.663293 6192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "065727ab9d7f4b61bda3adb6a82a66ac"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "931ffd99a6d180e27a02a2b8c6980911"
server_key_iv: "9366a029e1e60e1562db7e9e8c117519"
server_key_version: "encryptionkey@0"
I20260809 15:01:52.663448 6192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:52.690793 6192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:52.694409 6192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:52.694633 6192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:52.695482 6192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:52.696697 6192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:52.696770 6192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:52.696854 6192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:52.696897 6192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:52.726902 6192 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:35779
I20260809 15:01:52.726963 6391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:35779 every 8 connection(s)
I20260809 15:01:52.728049 6192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:52.736558 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 6192
I20260809 15:01:52.736742 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:01:52.737298 28394 external_mini_cluster.cc:1428] Setting key b935d7b38cfbaac850288892ecb2233b
Aug 09 15:01:52 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287712, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:01:52.746615 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:52.730475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:55919 (local address 127.27.186.190:36429)
0809 15:01:52.730634 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:52.730638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:52.731673 (+ 1035us) server_negotiation.cc:407] Connection header received
0809 15:01:52.733363 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:52.733367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:52.733436 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:52.733520 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:52.734916 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.735647 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.737552 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.737950 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.738084 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:52.741299 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:52.741304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:52.741307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:52.741339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:52.743747 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:52.744437 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:52.744442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:52.744444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:52.744573 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:52.744978 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:52.744983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:52.744987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:52.745244 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:52.745377 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:52.746042 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 15:01:52.746276 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:01:52.747498 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:52.730843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:55919)
0809 15:01:52.731511 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:52.731563 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:52.733116 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:52.733694 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:52.733701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:52.734223 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:52.734761 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:52.734778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.735758 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.735761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:52.737414 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:52.737430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:52.738183 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:52.738191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:52.738413 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:52.739068 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:52.739090 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:52.741098 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:52.743893 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:52.743902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:52.743917 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:52.744327 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:52.744676 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:52.744681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:52.744685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:52.744874 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:52.745368 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:52.745375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:52.745670 (+ 295us) client_negotiation.cc:769] Sending connection context
0809 15:01:52.745900 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 15:01:52.746431 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":288,"threads_started":1}
I20260809 15:01:52.748811 6392 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36429
I20260809 15:01:52.749082 6392 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:52.749780 6392 heartbeater.cc:511] Master 127.27.186.190:36429 requested a full tablet report, sending...
I20260809 15:01:52.751245 5855 ts_manager.cc:194] Registered new tserver with Master: 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779)
I20260809 15:01:52.752262 5855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:55919
WARNING: no policy specified for oryx/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab.
Entry for principal oryx/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab.
I20260809 15:01:52.782193 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36429
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:53.109061 6397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:53.109329 6397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:53.109454 6397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:53.116583 6397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:53.116691 6397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:53.116851 6397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:53.126178 6397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:53.127276 6397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:53.128523 6397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:53.141212 6407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:53.141295 6412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:53.141327 6408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:53.142350 6397 server_base.cc:1034] running on GCE node
I20260809 15:01:53.142908 6397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:53.143793 6397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:53.145016 6397 hybrid_clock.cc:648] HybridClock initialized: now 1786287713144974 us; error 29 us; skew 500 ppm
I20260809 15:01:53.145306 6397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287713, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:53.148218 6397 init.cc:377] Logged in from keytab as oryx/127.27.186.131@KRBTEST.COM (short username oryx)
I20260809 15:01:53.149541 6397 webserver.cc:460] Webserver started at http://127.27.186.131:34313/ using document root <none> and password file <none>
I20260809 15:01:53.150199 6397 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:53.150326 6397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:53.150599 6397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:53.153283 6397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "c021a3a60d38450381b12668e655c4df"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "ce8dbf18b19ded4a618f0f18cef181d1"
server_key_iv: "2ccf77c9c95050b73aedd92cfcecabb1"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.154039 6397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "c021a3a60d38450381b12668e655c4df"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "ce8dbf18b19ded4a618f0f18cef181d1"
server_key_iv: "2ccf77c9c95050b73aedd92cfcecabb1"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.158994 6397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 15:01:53.162178 6422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:53.163369 6397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:01:53.163513 6397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "c021a3a60d38450381b12668e655c4df"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "ce8dbf18b19ded4a618f0f18cef181d1"
server_key_iv: "2ccf77c9c95050b73aedd92cfcecabb1"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.163682 6397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:53.185138 6397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:53.188757 6397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:53.188999 6397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:53.189845 6397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:53.191057 6397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:53.191139 6397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:53.191255 6397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:53.191313 6397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:53.220818 6397 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:35347
I20260809 15:01:53.220882 6594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:35347 every 8 connection(s)
I20260809 15:01:53.221939 6397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:53.229056 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 6397
I20260809 15:01:53.229229 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:01:53.229856 28394 external_mini_cluster.cc:1428] Setting key e4a795329bb7c7604ba52532e4dbabfb
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287713, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:01:53.241060 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.224144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:46879 (local address 127.27.186.190:36429)
0809 15:01:53.224308 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.224313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.225349 (+ 1036us) server_negotiation.cc:407] Connection header received
0809 15:01:53.227092 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.227096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.227162 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.227244 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.228549 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.229644 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.231721 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.232309 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.232408 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.236082 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.236087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.236090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.236124 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.238642 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.239299 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.239302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.239304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.239413 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.239738 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.239741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.239743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.239995 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.240099 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.240662 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.240814 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:01:53.241972 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.224524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:46879)
0809 15:01:53.225201 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.225242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.226852 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.227412 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.227420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.227864 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.228398 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.228415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.229853 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.229858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.231559 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.231573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.232560 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.232569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.232857 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.233609 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.233635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.235879 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.238784 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.238791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.238802 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.239198 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.239486 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.239489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.239491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.239660 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.240081 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.240086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.240379 (+ 293us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.240587 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.240946 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":248,"threads_started":1}
I20260809 15:01:53.243240 6595 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36429
I20260809 15:01:53.243557 6595 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:53.244400 6595 heartbeater.cc:511] Master 127.27.186.190:36429 requested a full tablet report, sending...
I20260809 15:01:53.245828 5855 ts_manager.cc:194] Registered new tserver with Master: c021a3a60d38450381b12668e655c4df (127.27.186.131:35347)
I20260809 15:01:53.246631 5855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:46879
I20260809 15:01:53.257086 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:01:53.277997 6180 heartbeater.cc:503] Master 127.27.186.190:36429 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.27.186.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.189.keytab.
Entry for principal oryx/127.27.186.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.189.keytab.
I20260809 15:01:53.303210 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:45513
--webserver_interface=127.27.186.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.189.keytab
--principal=oryx/127.27.186.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:36429,127.27.186.189:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:53.613862 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:53.614142 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:53.614291 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:53.621208 6602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:53.621299 6602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:53.621349 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:53.621407 6602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:53.621449 6602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:53.631793 6602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:36429,127.27.186.189:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.189.keytab
--principal=oryx/127.27.186.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.186.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
W20260809 15:01:53.632143 6602 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.186.190:36429,127.27.186.189:45513'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 15:01:53.633302 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:53.634747 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:53.646083 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:53.646049 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:53.646083 6615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:53.646696 6602 server_base.cc:1034] running on GCE node
I20260809 15:01:53.647424 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:53.648618 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:53.649825 6602 hybrid_clock.cc:648] HybridClock initialized: now 1786287713649756 us; error 45 us; skew 500 ppm
I20260809 15:01:53.650161 6602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287713, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:53.656677 6602 init.cc:377] Logged in from keytab as oryx/127.27.186.189@KRBTEST.COM (short username oryx)
I20260809 15:01:53.658066 6602 webserver.cc:460] Webserver started at http://127.27.186.189:35335/ using document root <none> and password file <none>
I20260809 15:01:53.658681 6602 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:53.658764 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:53.659009 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:53.661834 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/instance:
uuid: "e32d364ca0db419a8d52ccc495486ec6"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "896c2e094c7766a32114cfad74d18e9a"
server_key_iv: "8b46b0bef7fa5fa33b82206c6190f515"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.662602 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal/instance:
uuid: "e32d364ca0db419a8d52ccc495486ec6"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "896c2e094c7766a32114cfad74d18e9a"
server_key_iv: "8b46b0bef7fa5fa33b82206c6190f515"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.667364 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 15:01:53.670825 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:53.672139 6602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 15:01:53.672302 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
uuid: "e32d364ca0db419a8d52ccc495486ec6"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "896c2e094c7766a32114cfad74d18e9a"
server_key_iv: "8b46b0bef7fa5fa33b82206c6190f515"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.672487 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:53.691540 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:53.694926 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:53.695145 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:53.711736 6602 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.189:45513
I20260809 15:01:53.711767 6713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.189:45513 every 8 connection(s)
I20260809 15:01:53.712888 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
I20260809 15:01:53.716234 6714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:53.720182 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 6602
I20260809 15:01:53.722697 6714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287713, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.189@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
I20260809 15:01:53.741107 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.726330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.27.186.190:36429)
0809 15:01:53.726527 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.726532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.727038 (+ 506us) server_negotiation.cc:407] Connection header received
0809 15:01:53.727306 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.727313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.727393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.727502 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.728933 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.730043 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.731685 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.732292 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.732414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.735402 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.735409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.735413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.735454 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.738117 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.738873 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.738877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.738879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.739005 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.739450 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.739454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.739457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.739827 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.739970 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.740598 (+ 628us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.740803 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 15:01:53.741990 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.722193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.27.186.189:45513)
0809 15:01:53.722931 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.722941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.722979 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:01:53.724565 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.724586 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.724948 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.725228 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.727036 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.728838 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.730327 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.731414 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.731579 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.735931 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.735938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.735947 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.735973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.738658 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.739312 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.739317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.739319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.739457 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.739827 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.739832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.739835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.740220 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.740434 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.740855 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.741063 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":539,"thread_start_us":324,"threads_started":1}
I20260809 15:01:53.742009 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.726262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:47146)
0809 15:01:53.726911 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.726952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.727151 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.727571 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.727578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.728360 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.728751 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.728767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.730219 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.730225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.731517 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.731532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.732481 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.732486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.732605 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.733148 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.733167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.735212 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.738239 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.738250 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.738260 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.738759 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.739109 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.739113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.739118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.739339 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.739947 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.739952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.740240 (+ 288us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.740517 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.740946 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":2,"spinlock_wait_cycles":10624,"thread_start_us":283,"threads_started":1}
I20260809 15:01:53.745375 6714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287713, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.189@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
I20260809 15:01:53.754576 6392 heartbeater.cc:503] Master 127.27.186.190:36429 was elected leader, sending a full tablet report...
I20260809 15:01:53.756165 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.746084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:34526)
0809 15:01:53.746232 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.746246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.746324 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.746520 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.746523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.746794 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.746915 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.746921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.747742 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.747745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.748550 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.748560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.749146 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.749150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.749300 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.749689 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.749703 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.751499 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.753979 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.753982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.753985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.754345 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.754712 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.754717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.754721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.754863 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.755337 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.755341 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.755492 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.755623 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.755881 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 15:01:53.756328 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.746102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.27.186.189:45513)
0809 15:01:53.746245 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.746250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.746286 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:01:53.746386 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.746389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.746434 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.746499 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.747018 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.747628 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.748664 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.749035 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.749129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.751651 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.751655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.751658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.751684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.753878 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.754445 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.754450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.754452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.754596 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.754979 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.754983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.754985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.755230 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.755332 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.755701 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.756022 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":48}
I20260809 15:01:53.760390 6714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Bootstrap starting.
I20260809 15:01:53.763141 6714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:53.764063 6714 log.cc:824] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:53.766366 6714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: No bootstrap required, opened a new log
I20260809 15:01:53.772989 6714 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } }
I20260809 15:01:53.773248 6714 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:53.773352 6714 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32d364ca0db419a8d52ccc495486ec6, State: Initialized, Role: FOLLOWER
I20260809 15:01:53.773860 6714 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } }
I20260809 15:01:53.774614 6722 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } } }
I20260809 15:01:53.774920 6722 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:53.775501 6714 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 15:01:53.783078 6742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:01:53.783217 6742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:01:53.786446 6602 master_runner.cc:428] Detected that this master e32d364ca0db419a8d52ccc495486ec6 is joining an existing cluster
I20260809 15:01:53.786542 6602 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.186.189:45513
I20260809 15:01:53.805980 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.790554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.27.186.189:45513)
0809 15:01:53.790707 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.790712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.792302 (+ 1590us) server_negotiation.cc:407] Connection header received
0809 15:01:53.792420 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.792424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.792475 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.792562 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.793544 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.794275 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.796339 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.797010 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.797101 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.799551 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.799557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.799562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.799594 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.803075 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.803869 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.803875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.803878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.804027 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.804452 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.804456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.804460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.804804 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.804940 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.805430 (+ 490us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.805642 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:01:53.806519 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.790589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.27.186.190:36429)
0809 15:01:53.790744 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.790749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.791649 (+ 900us) server_negotiation.cc:407] Connection header received
0809 15:01:53.791839 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.791845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.791921 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.792032 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.793551 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.794685 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.796808 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.797387 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.797469 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.799943 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.799950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.799954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.799992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.803140 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.804150 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.804155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.804158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.804326 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.804782 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.804787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.804790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.805114 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.805236 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.805875 (+ 639us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.806190 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:01:53.807122 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.790860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:34536)
0809 15:01:53.792212 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.792227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.792335 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.792594 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.792597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.793117 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.793420 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.793429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.794444 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.794448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.796172 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.796187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.797379 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.797386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.797571 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.798192 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.798224 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.799369 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.803225 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.803230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.803234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.803753 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.804154 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.804159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.804162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.804345 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.805021 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.805026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.805163 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.806307 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.806573 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"mutex_wait_us":3,"thread_start_us":358,"threads_started":1}
I20260809 15:01:53.807919 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.790414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:47152)
0809 15:01:53.791534 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.791553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.791688 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.792223 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.792228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.792908 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.793367 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.793385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.794873 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.794878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.796620 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.796639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.797727 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.797733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.797922 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.798494 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.798516 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.799708 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.803555 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.803560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.803563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.804017 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.804473 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.804477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.804480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.804649 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.805352 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.805356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.805564 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.806746 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.807076 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":25,"thread_start_us":1512,"threads_started":1}
Aug 09 15:01:53 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.190@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.186.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.188@KRBTEST.COM" created.
I20260809 15:01:53.829946 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.813791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.27.186.189:45513)
0809 15:01:53.813921 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.813925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.814709 (+ 784us) server_negotiation.cc:407] Connection header received
0809 15:01:53.814989 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.814994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.815090 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.815198 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.816451 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.817511 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.819341 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.820127 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.820265 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.824086 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.824093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.824098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.824137 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.826937 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.827811 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.827817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.827820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.827976 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.828405 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.828410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.828413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.828721 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.828871 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.829408 (+ 537us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.829631 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:01:53.830089 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.813761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:34540)
0809 15:01:53.814408 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.814614 (+ 206us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.814786 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.815261 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.815268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.816058 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.816289 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.816299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.817660 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.817666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.819187 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.819202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.820311 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.820317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.820512 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.821140 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.821163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.823892 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.827084 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.827096 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.827103 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.827656 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.828090 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.828095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.828099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.828279 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.828873 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.828881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.829066 (+ 185us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.829282 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.829602 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":295,"threads_started":1}
I20260809 15:01:53.831909 5855 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.186.189:45513: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6"
member_type: NON_VOTER
last_known_addr {
host: "127.27.186.189"
port: 45513
}
attrs {
promote: true
}
}
dest_uuid: "8df908b68a034b08b5862e491b99c8a9"
cas_config_opid_index: -1
I20260809 15:01:53.833406 5855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } }
I20260809 15:01:53.841355 6681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:53.845118 6681 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Refusing update from remote peer 8df908b68a034b08b5862e491b99c8a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 15:01:53.849246 6760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:53.849164 6759 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.188.keytab.
Entry for principal oryx/127.27.186.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.188.keytab.
I20260809 15:01:53.856474 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:40181
--webserver_interface=127.27.186.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.188.keytab
--principal=oryx/127.27.186.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:01:53.863333 6759 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.186.189:45513
I20260809 15:01:53.864367 6602 master.cc:537] Master@127.27.186.189:45513 shutting down...
I20260809 15:01:53.867450 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:53.867944 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:53.899592 6681 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:53.904112 6778 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:53.904417 6722 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df908b68a034b08b5862e491b99c8a9. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:53.904589 6778 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:53.904886 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:53.905229 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:53.904654 6722 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:53.908152 6602 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 LEARNER]: Raft consensus shutting down.
I20260809 15:01:53.908620 6602 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 LEARNER]: Raft consensus is shut down!
I20260809 15:01:53.908776 6602 tablet_replica.cc:330] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: stopping tablet replica
I20260809 15:01:53.920845 6602 master.cc:552] Master@127.27.186.189:45513 shutdown complete.
I20260809 15:01:53.920976 6602 master_runner.cc:305] Clearing existing system tablet
I20260809 15:01:53.921464 6602 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:53.921571 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:53.924527 6602 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:01:53.926237 6781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:53.926542 6602 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:53.926823 6602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 15:01:53.926914 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
uuid: "e32d364ca0db419a8d52ccc495486ec6"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "896c2e094c7766a32114cfad74d18e9a"
server_key_iv: "8b46b0bef7fa5fa33b82206c6190f515"
server_key_version: "encryptionkey@0"
I20260809 15:01:53.931674 6602 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:53.938432 6602 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 15:01:53.938602 6602 log.cc:1196] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 15:01:53.939085 6602 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Deleting consensus metadata
I20260809 15:01:53.939378 6602 master_runner.cc:315] Copying system tablet from 127.27.186.190:36429
I20260809 15:01:53.942243 6602 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.186.190:36429
I20260809 15:01:53.964586 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.943227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.27.186.190:36429)
0809 15:01:53.943408 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:53.943415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:53.950587 (+ 7172us) server_negotiation.cc:407] Connection header received
0809 15:01:53.950776 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:53.950782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:53.950882 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:53.950998 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:53.952010 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.953104 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.955042 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.955696 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.955822 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.958097 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:53.958104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:53.958109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:53.958150 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:53.961589 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.962427 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.962433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.962437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.962583 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:53.963003 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:53.963008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:53.963011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:53.963374 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:53.963529 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:53.964002 (+ 473us) server_negotiation.cc:299] Negotiation successful
0809 15:01:53.964246 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:01:53.964643 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:53.943141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:47156)
0809 15:01:53.950453 (+ 7312us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:53.950476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:53.950641 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:53.951039 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:53.951044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:53.951570 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:53.951851 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.951865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.953292 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.953297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.954860 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:53.954876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:53.955870 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:53.955876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:53.956072 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:53.956816 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:53.956839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:53.957866 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:53.961732 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.961738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.961742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.962314 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.962714 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:53.962719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:53.962722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:53.962884 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:53.963502 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:53.963507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:53.963708 (+ 201us) client_negotiation.cc:769] Sending connection context
0809 15:01:53.963864 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 15:01:53.964208 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7174,"thread_start_us":7039,"threads_started":1}
I20260809 15:01:53.965791 5888 tablet_copy_service.cc:140] P 8df908b68a034b08b5862e491b99c8a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e32d364ca0db419a8d52ccc495486ec6 ({username='oryx', principal='oryx/127.27.186.189@KRBTEST.COM'} at 127.0.0.1:47156)
I20260809 15:01:53.966301 5888 tablet_copy_service.cc:161] P 8df908b68a034b08b5862e491b99c8a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e32d364ca0db419a8d52ccc495486ec6 at {username='oryx', principal='oryx/127.27.186.189@KRBTEST.COM'} at 127.0.0.1:47156: session id = e32d364ca0db419a8d52ccc495486ec6-00000000000000000000000000000000
I20260809 15:01:53.971242 5888 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:01:53.973820 6602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 15:01:53.980650 6602 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: tablet copy: Starting download of 0 data blocks...
I20260809 15:01:53.981074 6602 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: tablet copy: Starting download of 1 WAL segments...
I20260809 15:01:53.984849 6602 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:01:53.989487 5888 tablet_copy_service.cc:342] P 8df908b68a034b08b5862e491b99c8a9: Request end of tablet copy session e32d364ca0db419a8d52ccc495486ec6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.186.189@KRBTEST.COM'} at 127.0.0.1:47156
I20260809 15:01:53.989732 5888 tablet_copy_service.cc:433] P 8df908b68a034b08b5862e491b99c8a9: ending tablet copy session e32d364ca0db419a8d52ccc495486ec6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e32d364ca0db419a8d52ccc495486ec6
I20260809 15:01:54.000329 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:54.002657 6794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:54.007210 6795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:54.014814 6602 server_base.cc:1034] running on GCE node
W20260809 15:01:54.015208 6799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:54.015583 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:54.016132 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:54.017321 6602 hybrid_clock.cc:648] HybridClock initialized: now 1786287714017263 us; error 35 us; skew 500 ppm
I20260809 15:01:54.017499 6602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287714, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:54.019356 6602 init.cc:377] Logged in from keytab as oryx/127.27.186.189@KRBTEST.COM (short username oryx)
I20260809 15:01:54.020481 6602 webserver.cc:460] Webserver started at http://127.27.186.189:37703/ using document root <none> and password file <none>
I20260809 15:01:54.020788 6602 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:54.020874 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:54.023154 6602 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:54.024684 6811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:54.025059 6602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 15:01:54.025172 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
uuid: "e32d364ca0db419a8d52ccc495486ec6"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "896c2e094c7766a32114cfad74d18e9a"
server_key_iv: "8b46b0bef7fa5fa33b82206c6190f515"
server_key_version: "encryptionkey@0"
I20260809 15:01:54.025322 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:54.040395 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:54.043527 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:54.064541 6602 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.189:45513
I20260809 15:01:54.064569 6888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.189:45513 every 8 connection(s)
I20260809 15:01:54.065375 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
I20260809 15:01:54.071470 6890 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:01:54.075706 6890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Bootstrap starting.
I20260809 15:01:54.088574 6890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:01:54.089025 6890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: Bootstrap complete.
I20260809 15:01:54.089792 6890 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } }
I20260809 15:01:54.089985 6890 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e32d364ca0db419a8d52ccc495486ec6, State: Initialized, Role: LEARNER
I20260809 15:01:54.090266 6890 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } }
I20260809 15:01:54.091037 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.091354 6890 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:54.091481 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:54.106149 6908 catalog_manager.cc:1259] Loaded cluster ID: 8450f063ff444687a284745672ea6993
I20260809 15:01:54.106231 6908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: loading cluster ID for follower catalog manager: success
I20260809 15:01:54.108568 6908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: acquiring CA information for follower catalog manager: success
I20260809 15:01:54.110569 6908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 15:01:54.244239 6759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.244590 6759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:54.250769 6595 heartbeater.cc:503] Master 127.27.186.190:36429 was elected leader, sending a full tablet report...
I20260809 15:01:54.253752 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.244667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:34544)
0809 15:01:54.244957 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.244979 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.245129 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.246409 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.246414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.246448 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:54.246642 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.246653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.248224 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.248231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.250349 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.250368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.252952 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.252958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.253170 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.253239 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.253347 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.253492 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260809 15:01:54.254127 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.245175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.27.186.189:45513)
0809 15:01:54.246033 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.246041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.246066 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:54.246141 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.246145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.246219 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.246339 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:54.246816 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.248062 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.251225 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.252740 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.252948 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.253160 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.253393 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.253544 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":509,"threads_started":1}
I20260809 15:01:54.255191 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df908b68a034b08b5862e491b99c8a9. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.255473 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:54.256170 6760 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: attempting to promote NON_VOTER e32d364ca0db419a8d52ccc495486ec6 to VOTER
I20260809 15:01:54.257118 6760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } }
I20260809 15:01:54.259017 6853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 LEARNER]: Refusing update from remote peer 8df908b68a034b08b5862e491b99c8a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 15:01:54.259682 6759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:01:54.262941 6760 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } }
I20260809 15:01:54.267201 6853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } }
I20260809 15:01:54.268158 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } }
I20260809 15:01:54.268195 6759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } }
I20260809 15:01:54.268389 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:54.268468 6759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:54.272253 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } }
I20260809 15:01:54.272483 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 15:01:54.279876 6768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:54.280180 6768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:54.280320 6768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:54.287523 6768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:01:54.287609 6768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:54.287653 6768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:54.287709 6768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:01:54.287761 6768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:01:54.296975 6768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.188.keytab
--principal=oryx/127.27.186.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:40181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.186.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:54.298095 6768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:54.299224 6768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:54.310005 6918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:54.310048 6924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:54.310204 6920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:54.311079 6768 server_base.cc:1034] running on GCE node
I20260809 15:01:54.311800 6768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:54.313030 6768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:54.314204 6768 hybrid_clock.cc:648] HybridClock initialized: now 1786287714314152 us; error 43 us; skew 500 ppm
I20260809 15:01:54.314498 6768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287714, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:54.317421 6768 init.cc:377] Logged in from keytab as oryx/127.27.186.188@KRBTEST.COM (short username oryx)
I20260809 15:01:54.318812 6768 webserver.cc:460] Webserver started at http://127.27.186.188:45703/ using document root <none> and password file <none>
I20260809 15:01:54.319423 6768 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:54.319528 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:54.319801 6768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:01:54.322582 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/instance:
uuid: "525c52b6bdc2459db16a822408954503"
format_stamp: "Formatted at 2026-08-09 15:01:54 on dist-test-slave-q9h9"
server_key: "3a6485331e3684ba044b659ae726d131"
server_key_iv: "94d905042e4c6e27018e4823d19ba283"
server_key_version: "encryptionkey@0"
I20260809 15:01:54.323372 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal/instance:
uuid: "525c52b6bdc2459db16a822408954503"
format_stamp: "Formatted at 2026-08-09 15:01:54 on dist-test-slave-q9h9"
server_key: "3a6485331e3684ba044b659ae726d131"
server_key_iv: "94d905042e4c6e27018e4823d19ba283"
server_key_version: "encryptionkey@0"
I20260809 15:01:54.328547 6768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 15:01:54.331828 6934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:54.332959 6768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:01:54.333107 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
uuid: "525c52b6bdc2459db16a822408954503"
format_stamp: "Formatted at 2026-08-09 15:01:54 on dist-test-slave-q9h9"
server_key: "3a6485331e3684ba044b659ae726d131"
server_key_iv: "94d905042e4c6e27018e4823d19ba283"
server_key_version: "encryptionkey@0"
I20260809 15:01:54.333281 6768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:54.354787 6768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:54.358635 6768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:54.358863 6768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:54.375619 6768 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.188:40181
I20260809 15:01:54.375662 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.188:40181 every 8 connection(s)
I20260809 15:01:54.376792 6768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
I20260809 15:01:54.377911 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 6768
I20260809 15:01:54.381001 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:54.389148 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.188@KRBTEST.COM
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287714, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.188@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:01:54.401657 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.380069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.27.186.188:40181)
0809 15:01:54.380810 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.380825 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.380873 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:01:54.383022 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.383055 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.383488 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.384080 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.385451 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.387420 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.388646 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.389614 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.389803 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.394435 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.394446 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.394459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.394506 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.397802 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.398597 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.398603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.398608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.398771 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.399206 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.399210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.399217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.399717 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.400009 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.400465 (+ 456us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.400807 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":318,"threads_started":1}
I20260809 15:01:54.406082 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.391572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:47160)
0809 15:01:54.392259 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.392308 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.392519 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.392886 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.392893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.393682 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.393990 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.394003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.395369 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.395372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.396629 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.396641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.397637 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.397641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.397752 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.398333 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.398362 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.400524 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.403815 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.403823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.403828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.404420 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.404756 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.404759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.404761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.404883 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.405300 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.405304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.405439 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.405584 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.405779 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":349,"threads_started":1}
I20260809 15:01:54.406114 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.391799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.27.186.190:36429)
0809 15:01:54.391989 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.391995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.392462 (+ 467us) server_negotiation.cc:407] Connection header received
0809 15:01:54.392657 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.392661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.392724 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.392799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.394153 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.395219 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.396780 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.397468 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.397602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.400688 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.400694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.400698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.400736 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.403697 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.404528 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.404532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.404534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.404664 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.404965 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.404968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.404970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.405211 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.405315 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.405654 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.405819 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:01:54.407362 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287714, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.188@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
I20260809 15:01:54.419240 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.407877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:34556)
0809 15:01:54.408030 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.408046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.408133 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.408428 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.408432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.408777 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.408899 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.408907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.409857 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.409862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.410841 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.410861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.411660 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.411666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.411904 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.412451 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.412471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.414681 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.417337 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.417340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.417342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.417684 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.417972 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.417975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.417978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.418132 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.418607 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.418609 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.418751 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.418868 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.419046 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 15:01:54.419363 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.408004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.27.186.189:45513)
0809 15:01:54.408184 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.408189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.408206 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:01:54.408257 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.408260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.408317 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.408408 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.409033 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.409735 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.410984 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.411502 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.411618 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.414868 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.414872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.414875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.414906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.417221 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.417767 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.417770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.417772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.417897 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.418214 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.418217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.418219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.418523 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.418633 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.418955 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.419124 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:01:54.420150 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287714, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.188@KRBTEST.COM for oryx/127.27.186.188@KRBTEST.COM
I20260809 15:01:54.430869 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.420825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.0.0.1:58036)
0809 15:01:54.420985 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.421001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.421089 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.421334 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.421339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.421646 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.421775 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.421784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.422660 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.422665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.423446 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.423456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.424084 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.424089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.424253 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.424671 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.424685 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.426459 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.428777 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.428782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.428786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.429101 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.429415 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.429420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.429424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.429557 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.430001 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.430005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.430165 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.430280 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.430506 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 15:01:54.431032 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.420798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.27.186.188:40181)
0809 15:01:54.420952 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.420957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.421053 (+ 96us) server_negotiation.cc:407] Connection header received
0809 15:01:54.421175 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.421179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.421224 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.421295 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.421894 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.422549 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.423566 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.423948 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.424046 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.426607 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.426610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.426613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.426638 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.428663 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.429200 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.429205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.429208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.429318 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.429650 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.429655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.429658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.429897 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.430055 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.430357 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.430658 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":47}
I20260809 15:01:54.435177 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Bootstrap starting.
I20260809 15:01:54.438107 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:54.439026 7029 log.cc:824] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:54.441256 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: No bootstrap required, opened a new log
I20260809 15:01:54.447968 7029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } }
I20260809 15:01:54.448237 7029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:54.448330 7029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525c52b6bdc2459db16a822408954503, State: Initialized, Role: FOLLOWER
I20260809 15:01:54.448808 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } }
I20260809 15:01:54.449572 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } } }
I20260809 15:01:54.449873 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:54.450436 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:54.456105 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 5960
W20260809 15:01:54.459652 7061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:01:54.459769 7061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:01:54.464885 6768 master_runner.cc:428] Detected that this master 525c52b6bdc2459db16a822408954503 is joining an existing cluster
I20260809 15:01:54.465049 6768 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.186.188:40181
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.129@KRBTEST.COM
I20260809 15:01:54.478302 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.457650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.27.186.129:36657)
0809 15:01:54.459264 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.459273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.459301 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:01:54.459368 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.459376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.459615 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.460764 (+ 1149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.461353 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.462662 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.464488 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.465303 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.465415 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.469932 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.469943 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.469954 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.469984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.472827 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.474491 (+ 1664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.474499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.474503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.474660 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.475556 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.475561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.475564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.475981 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.476262 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.477547 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.477831 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":2798,"threads_started":1}
I20260809 15:01:54.480744 6173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:54.498585 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.474373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.27.186.190:36429)
0809 15:01:54.474756 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.474761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.474779 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:01:54.474854 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.474859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.474935 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.475047 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.476324 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.477419 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.479453 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.480130 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.480228 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.491887 (+ 11659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.491893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.491898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.491936 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.495407 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.496247 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.496252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.496255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.496432 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.496940 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.496945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.496948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.497280 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.497408 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.498004 (+ 596us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.498229 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260809 15:01:54.499779 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.473864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:47176)
0809 15:01:54.474548 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.474566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.474686 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.475090 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.475095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.475731 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.476131 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.476144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.477594 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.477598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.479286 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.479301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.486533 (+ 7232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.486539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.486745 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.488055 (+ 1310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.488076 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.491627 (+ 3551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.495590 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.495594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.495598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.496095 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.496588 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.496593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.496596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.496808 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.497494 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.497499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.497706 (+ 207us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.499024 (+ 1318us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.499334 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":875,"threads_started":1}
I20260809 15:01:54.507203 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.473950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:34564)
0809 15:01:54.482672 (+ 8722us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.482696 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.483079 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.483602 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.483607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.484299 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.484697 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.484711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.486260 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.486264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.487851 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.487864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.491207 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.491218 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.491384 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.493240 (+ 1856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.493267 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.494430 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.499322 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.499326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.499329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.499740 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.500571 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.500575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.500578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.500770 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.506152 (+ 5382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.506157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.506351 (+ 194us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.506478 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.506780 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8589,"mutex_wait_us":22,"thread_start_us":12528,"threads_started":1}
I20260809 15:01:54.507815 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.474363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.27.186.189:45513)
0809 15:01:54.474557 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.474563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.482781 (+ 8218us) server_negotiation.cc:407] Connection header received
0809 15:01:54.483245 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.483251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.483327 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.483393 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.484890 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.485993 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.487986 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.488391 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.488494 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.494627 (+ 6133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.494633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.494637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.494675 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.498221 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.500250 (+ 2029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.500255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.500258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.500417 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.505549 (+ 5132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.505554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.505557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.505872 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.505972 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.507267 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.507499 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:01:54.515288 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.476042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.0.0.1:58046)
0809 15:01:54.486376 (+ 10334us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.486397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.486761 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.487186 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.487193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.487827 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.491233 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.491248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.492967 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.492972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.494689 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.494703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.495816 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.495821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.495981 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.498427 (+ 2446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.498453 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.500045 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.504934 (+ 4889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.504938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.504941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.505424 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.513536 (+ 8112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.513539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.513541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.513660 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.514366 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.514369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.514481 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.514585 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.514861 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10205,"thread_start_us":6751,"threads_started":1}
I20260809 15:01:54.515332 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.475490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.27.186.188:40181)
0809 15:01:54.475667 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.475672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.486470 (+ 10798us) server_negotiation.cc:407] Connection header received
0809 15:01:54.486888 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.486896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.486954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.487010 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.491384 (+ 4374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.492467 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.494824 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.495468 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.495552 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.500983 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.500988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.500992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.501024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.504784 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.508190 (+ 3406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.508195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.508199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.508362 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.513780 (+ 5418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.513786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.513790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.514164 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.514328 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.514713 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.514996 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":61}
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.190@KRBTEST.COM for oryx/127.27.186.188@KRBTEST.COM
I20260809 15:01:54.529486 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.506911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.27.186.188:40181)
0809 15:01:54.514423 (+ 7512us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.514429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.514450 (+ 21us) server_negotiation.cc:407] Connection header received
0809 15:01:54.514528 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.514533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.514607 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.515545 (+ 938us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.515834 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.517106 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.518882 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.519638 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.519762 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.523302 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.523309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.523314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.523347 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.526758 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.527485 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.527490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.527494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.527637 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.528020 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.528025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.528028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.528359 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.528494 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.528919 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.529105 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7397,"thread_start_us":7253,"threads_started":1}
I20260809 15:01:54.529580 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.506280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.0.0.1:58054)
0809 15:01:54.506618 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.506633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.506726 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.514774 (+ 8048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.514779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.515208 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.515408 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.515419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.517255 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.517261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.518684 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.518699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.519797 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.519804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.519970 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.520478 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.520498 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.523070 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.526872 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.526877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.526881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.527346 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.527744 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.527749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.527753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.527909 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.528483 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.528489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.528652 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.528813 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.529072 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260809 15:01:54.531320 5855 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.186.188:40181: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "525c52b6bdc2459db16a822408954503"
member_type: NON_VOTER
last_known_addr {
host: "127.27.186.188"
port: 40181
}
attrs {
promote: true
}
}
dest_uuid: "8df908b68a034b08b5862e491b99c8a9"
cas_config_opid_index: 6
I20260809 15:01:54.532040 5855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } }
I20260809 15:01:54.534583 6853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Refusing update from remote peer 8df908b68a034b08b5862e491b99c8a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 15:01:54.535071 6985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:54.535218 6759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:01:54.538810 6985 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Refusing update from remote peer 8df908b68a034b08b5862e491b99c8a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 15:01:54.539947 6759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:54.540567 6760 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 525c52b6bdc2459db16a822408954503 (127.27.186.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.544759 6853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 525c52b6bdc2459db16a822408954503 (127.27.186.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.548516 6760 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.186.188:40181
I20260809 15:01:54.549062 6768 master.cc:537] Master@127.27.186.188:40181 shutting down...
I20260809 15:01:54.555233 6985 raft_consensus.cc:858] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } }
W20260809 15:01:54.555485 6985 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } }
I20260809 15:01:54.555840 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.556100 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:54.556408 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.556620 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:54.557106 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.557374 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:54.579617 6985 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.584029 7091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.584434 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.584802 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 15:01:54.584712 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df908b68a034b08b5862e491b99c8a9. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: NON_VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: true } } }
I20260809 15:01:54.584923 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 15:01:54.584440 7091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 15:01:54.586774 6760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.587006 6760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
W20260809 15:01:54.587508 5835 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260809 15:01:54.587985 6768 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 15:01:54.588407 6768 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 15:01:54.588565 6768 tablet_replica.cc:330] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: stopping tablet replica
I20260809 15:01:54.603361 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.602859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.0.0.1:58062)
0809 15:01:54.603061 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.603204 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 15:01:54.603621 5835 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 -> Peer 525c52b6bdc2459db16a822408954503 (127.27.186.188:40181): Couldn't send request to peer 525c52b6bdc2459db16a822408954503. Status: Network error: Client connection negotiation failed: client connection to 127.27.186.188:40181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:01:54.606447 6768 master.cc:552] Master@127.27.186.188:40181 shutdown complete.
I20260809 15:01:54.606536 6768 master_runner.cc:305] Clearing existing system tablet
I20260809 15:01:54.606909 6768 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:54.607010 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:54.609956 6768 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:01:54.611722 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:54.612138 6768 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:54.612419 6768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 15:01:54.612531 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
uuid: "525c52b6bdc2459db16a822408954503"
format_stamp: "Formatted at 2026-08-09 15:01:54 on dist-test-slave-q9h9"
server_key: "3a6485331e3684ba044b659ae726d131"
server_key_iv: "94d905042e4c6e27018e4823d19ba283"
server_key_version: "encryptionkey@0"
I20260809 15:01:54.617192 6768 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:54.617326 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 5960
I20260809 15:01:54.628394 6768 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 15:01:54.628525 6768 log.cc:1196] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 15:01:54.628890 6768 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Deleting consensus metadata
I20260809 15:01:54.629088 6768 master_runner.cc:315] Copying system tablet from 127.27.186.190:36429
I20260809 15:01:54.638466 6768 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: tablet copy: Beginning tablet copy session from remote peer at address 127.27.186.190:36429
I20260809 15:01:54.653935 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:36657
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:36429
--webserver_port=43163
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:01:54.654448 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.639310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.27.186.190:36429)
0809 15:01:54.639593 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:54.639599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:54.639917 (+ 318us) server_negotiation.cc:407] Connection header received
0809 15:01:54.640191 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:54.640197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:54.640302 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:54.640379 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:54.641576 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.642813 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.644725 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.645398 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.645492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.647890 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:54.647897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:54.647902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:54.647942 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:54.651311 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.652083 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.652089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.652093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.652260 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:54.652745 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:54.652750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:54.652754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:54.653077 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:54.653208 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:54.653845 (+ 637us) server_negotiation.cc:299] Negotiation successful
0809 15:01:54.654123 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260809 15:01:54.654467 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:54.639280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:47190)
0809 15:01:54.639803 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:54.639827 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:54.640030 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:54.640591 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:54.640596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:54.641129 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:54.641401 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.641414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.643007 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.643012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.644560 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:54.644575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:54.645762 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:54.645767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:54.645952 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:54.646610 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:54.646633 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:54.647680 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:54.651461 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.651466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.651470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.651950 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.652418 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:54.652423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:54.652427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:54.652621 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:54.653316 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:54.653321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:54.653503 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 15:01:54.653701 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 15:01:54.654005 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260809 15:01:54.658849 5888 tablet_copy_service.cc:140] P 8df908b68a034b08b5862e491b99c8a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 525c52b6bdc2459db16a822408954503 ({username='oryx', principal='oryx/127.27.186.188@KRBTEST.COM'} at 127.0.0.1:47190)
I20260809 15:01:54.659013 5888 tablet_copy_service.cc:161] P 8df908b68a034b08b5862e491b99c8a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 525c52b6bdc2459db16a822408954503 at {username='oryx', principal='oryx/127.27.186.188@KRBTEST.COM'} at 127.0.0.1:47190: session id = 525c52b6bdc2459db16a822408954503-00000000000000000000000000000000
I20260809 15:01:54.661988 5888 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:01:54.663641 6768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 15:01:54.670403 6768 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: tablet copy: Starting download of 0 data blocks...
I20260809 15:01:54.670703 6768 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: tablet copy: Starting download of 1 WAL segments...
I20260809 15:01:54.672890 6768 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:01:54.676963 5888 tablet_copy_service.cc:342] P 8df908b68a034b08b5862e491b99c8a9: Request end of tablet copy session 525c52b6bdc2459db16a822408954503-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.186.188@KRBTEST.COM'} at 127.0.0.1:47190
I20260809 15:01:54.677067 5888 tablet_copy_service.cc:433] P 8df908b68a034b08b5862e491b99c8a9: ending tablet copy session 525c52b6bdc2459db16a822408954503-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 525c52b6bdc2459db16a822408954503
I20260809 15:01:54.690459 6768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:54.693009 7107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:54.699486 7108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:54.703645 7111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:54.704064 6768 server_base.cc:1034] running on GCE node
I20260809 15:01:54.704378 6768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:54.704973 6768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:54.706115 6768 hybrid_clock.cc:648] HybridClock initialized: now 1786287714706058 us; error 36 us; skew 500 ppm
I20260809 15:01:54.706357 6768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:54 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287714, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:54.708477 6768 init.cc:377] Logged in from keytab as oryx/127.27.186.188@KRBTEST.COM (short username oryx)
I20260809 15:01:54.713413 6768 webserver.cc:460] Webserver started at http://127.27.186.188:44317/ using document root <none> and password file <none>
I20260809 15:01:54.713763 6768 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:54.713873 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:54.716898 6768 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:01:54.718798 7124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:54.719257 6768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260809 15:01:54.719396 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
uuid: "525c52b6bdc2459db16a822408954503"
format_stamp: "Formatted at 2026-08-09 15:01:54 on dist-test-slave-q9h9"
server_key: "3a6485331e3684ba044b659ae726d131"
server_key_iv: "94d905042e4c6e27018e4823d19ba283"
server_key_version: "encryptionkey@0"
I20260809 15:01:54.719589 6768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:54.769634 6768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:54.773836 6768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:54.802955 6768 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.188:40181
I20260809 15:01:54.803018 7200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.188:40181 every 8 connection(s)
I20260809 15:01:54.804045 6768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
I20260809 15:01:54.809556 7203 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:01:54.813482 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Bootstrap starting.
I20260809 15:01:54.830057 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:01:54.830606 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Bootstrap complete.
I20260809 15:01:54.831733 7203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } }
I20260809 15:01:54.832008 7203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 525c52b6bdc2459db16a822408954503, State: Initialized, Role: LEARNER
I20260809 15:01:54.832258 7203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } }
I20260809 15:01:54.833420 7203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:01:54.833179 7207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:54.833774 7207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:54.843652 7232 catalog_manager.cc:1259] Loaded cluster ID: 8450f063ff444687a284745672ea6993
I20260809 15:01:54.843755 7232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: loading cluster ID for follower catalog manager: success
I20260809 15:01:54.847164 7232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: acquiring CA information for follower catalog manager: success
I20260809 15:01:54.849715 7232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 15:01:55.035255 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:55.035487 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:55.035583 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:55.042816 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:55.042910 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:55.043020 7106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:01:55.052260 7106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:36657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=43163
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:55.053325 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:55.054572 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:55.068578 7238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:55.070995 7241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:55.072688 7106 server_base.cc:1034] running on GCE node
W20260809 15:01:55.073575 7239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:55.074299 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:55.075248 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:55.076465 7106 hybrid_clock.cc:648] HybridClock initialized: now 1786287715076407 us; error 55 us; skew 500 ppm
I20260809 15:01:55.076761 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:55.079803 7106 init.cc:377] Logged in from keytab as oryx/127.27.186.129@KRBTEST.COM (short username oryx)
I20260809 15:01:55.081101 7106 webserver.cc:460] Webserver started at http://127.27.186.129:43163/ using document root <none> and password file <none>
I20260809 15:01:55.081779 7106 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:55.081910 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:55.087064 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 15:01:55.089663 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:55.091028 7106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:01:55.091184 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "cc9d3ce19fa54354a136d05df2e352d0"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "2eba85a0eb50505429549adfee7a660c"
server_key_iv: "be02c50d309d4ed0aaeb7a91eba0f88c"
server_key_version: "encryptionkey@0"
I20260809 15:01:55.092133 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:55.107103 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:55.110888 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:55.111152 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:55.112017 7106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:55.113327 7106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:55.113430 7106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:55.113528 7106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:55.113602 7106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:55.142829 7106 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:36657
I20260809 15:01:55.142890 7431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:36657 every 8 connection(s)
I20260809 15:01:55.144153 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:01:55.152449 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 7106
I20260809 15:01:55.152774 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 6192
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.188@KRBTEST.COM
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.130@KRBTEST.COM
I20260809 15:01:55.171134 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.147942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.27.186.129:57343)
0809 15:01:55.149728 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.149742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.151323 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.152027 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.152038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.152970 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:55.154116 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.154140 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.155940 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.155945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.157343 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.157355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.158392 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.158400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.158657 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.159432 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:55.159467 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:55.162433 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:55.166370 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.166379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.166393 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.167155 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.167593 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.167598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.167602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.167935 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.168628 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:55.168636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:55.169019 (+ 383us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.169317 (+ 298us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.169771 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"mutex_wait_us":144,"spinlock_wait_cycles":25600,"thread_start_us":1975,"threads_started":1}
I20260809 15:01:55.171818 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.148067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.27.186.129:36695)
0809 15:01:55.150359 (+ 2292us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.150372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.151324 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.152081 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.152085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.153140 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:55.154129 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.154140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.155630 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.155635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.158240 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.158254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.159348 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.159355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.159527 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.160096 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:55.160117 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:55.162807 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:55.168478 (+ 5671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.168483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.168487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.169026 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.169476 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.169481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.169485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.169656 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.170453 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:55.170457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:55.170601 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.170755 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.171012 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2198,"mutex_wait_us":112,"spinlock_wait_cycles":76032,"thread_start_us":1353,"threads_started":1}
I20260809 15:01:55.172261 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.147017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:57343 (local address 127.27.186.188:40181)
0809 15:01:55.147538 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.147542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.149825 (+ 2283us) server_negotiation.cc:407] Connection header received
0809 15:01:55.151663 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.151669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.151765 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.152582 (+ 817us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.154317 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.155758 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.157480 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.158129 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.159050 (+ 921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.162710 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.162716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.162721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.162761 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.166196 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.167315 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.167320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.167324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.167462 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.168087 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.168091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.168094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.168465 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.171566 (+ 3101us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.171695 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.171903 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":241,"threads_started":1}
I20260809 15:01:55.172400 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.147904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:35873)
0809 15:01:55.148960 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.149020 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.151319 (+ 2299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.152029 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.152039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.152946 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:55.154124 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.154141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.155436 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.155441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.157569 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.157586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.158686 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.158696 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.158865 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.160254 (+ 1389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:55.160287 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:55.164295 (+ 4008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:55.168263 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.168267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.168270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.168790 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.169203 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.169207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.169210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.170717 (+ 1507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.171442 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:55.171446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:55.171579 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.171702 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.171966 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":157,"spinlock_wait_cycles":3840,"thread_start_us":480,"threads_started":1}
I20260809 15:01:55.173208 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.146944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:35873 (local address 127.27.186.190:36429)
0809 15:01:55.147157 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.147164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.149145 (+ 1981us) server_negotiation.cc:407] Connection header received
0809 15:01:55.151594 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.151600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.151678 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.151760 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.154449 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.155284 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.157700 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.158179 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.158312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.164500 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.164506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.164510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.164543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.168125 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.168926 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.168931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.168934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.169070 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.170859 (+ 1789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.170864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.170866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.171227 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.171380 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.172727 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.172900 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:01:55.173611 7437 heartbeater.cc:347] Connected to a master server at 127.27.186.188:40181
I20260809 15:01:55.173637 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.147007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:36695 (local address 127.27.186.189:45513)
0809 15:01:55.147689 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.147695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.150459 (+ 2764us) server_negotiation.cc:407] Connection header received
0809 15:01:55.151663 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.151669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.151749 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.151849 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.154317 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.155478 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.158398 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.158974 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.159079 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.163637 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.163644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.163649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.163691 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.168317 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.169173 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.169179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.169182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.169340 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.169798 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.169803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.169807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.170227 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.170385 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.172798 (+ 2413us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.173013 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":688,"threads_started":1}
I20260809 15:01:55.173931 7437 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:55.174005 7438 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36429
I20260809 15:01:55.174157 7438 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:55.174798 7441 heartbeater.cc:347] Connected to a master server at 127.27.186.189:45513
I20260809 15:01:55.174952 7437 heartbeater.cc:511] Master 127.27.186.188:40181 requested a full tablet report, sending...
I20260809 15:01:55.174996 7438 heartbeater.cc:511] Master 127.27.186.190:36429 requested a full tablet report, sending...
I20260809 15:01:55.175305 7441 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:55.175796 7441 heartbeater.cc:511] Master 127.27.186.189:45513 requested a full tablet report, sending...
I20260809 15:01:55.177431 7147 ts_manager.cc:194] Registered new tserver with Master: cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657)
I20260809 15:01:55.177430 6841 ts_manager.cc:194] Registered new tserver with Master: cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657)
I20260809 15:01:55.177816 5855 ts_manager.cc:194] Re-registered known tserver with Master: cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657)
I20260809 15:01:55.178910 5855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:35873
I20260809 15:01:55.179441 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.154152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.27.186.130:35779)
0809 15:01:55.154763 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.154781 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.158631 (+ 3850us) server_negotiation.cc:407] Connection header received
0809 15:01:55.158920 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.158929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.159139 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.159278 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.161463 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.162824 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.164448 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.165074 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.165148 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.171021 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.171034 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.171049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.171094 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.176267 (+ 5173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.176951 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.176959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.176962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.177126 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.177630 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.177636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.177639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.178166 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.178454 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.178756 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.179070 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":1222,"threads_started":1}
I20260809 15:01:55.180517 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:55.180792 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:55.180876 6385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:55.189679 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.181200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.0.0.1:58074)
0809 15:01:55.182192 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.182217 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.182494 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.183035 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.183040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.183067 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:55.183273 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.183284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.184752 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.184759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.186938 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.186955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.188899 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.188906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.189088 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.189122 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.189208 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.189336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":655,"threads_started":1}
I20260809 15:01:55.190376 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.181629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.27.186.188:40181)
0809 15:01:55.181798 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.181804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.182604 (+ 800us) server_negotiation.cc:407] Connection header received
0809 15:01:55.182684 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.182689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.182785 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.182879 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:55.183453 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.184561 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.187123 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.188434 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.188601 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.188777 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.189955 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.190121 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:01:55.191139 7167 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260809 15:01:55.191624 7207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df908b68a034b08b5862e491b99c8a9. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: NON_VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: true } } }
I20260809 15:01:55.191913 7207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: LEARNER
I20260809 15:01:55.196204 7456 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: attempting to promote NON_VOTER 525c52b6bdc2459db16a822408954503 to VOTER
I20260809 15:01:55.196792 7456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } }
I20260809 15:01:55.198585 7167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 LEARNER]: Refusing update from remote peer 8df908b68a034b08b5862e491b99c8a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 15:01:55.198724 6853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Refusing update from remote peer 8df908b68a034b08b5862e491b99c8a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 15:01:55.199255 6758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 15:01:55.199499 7456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 15:01:55.201673 7456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 525c52b6bdc2459db16a822408954503 (127.27.186.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.203173 7167 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 525c52b6bdc2459db16a822408954503 (127.27.186.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.204481 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.204675 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:55.204878 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.205021 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:55.205338 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.205550 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:55.205821 6853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 525c52b6bdc2459db16a822408954503 (127.27.186.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.207289 7207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.207523 7207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:55.209044 7464 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8df908b68a034b08b5862e491b99c8a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:55.209386 7464 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:55.277288 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 6192
I20260809 15:01:55.297631 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:35779
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:36429
--webserver_port=32935
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:01:55.609589 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:55.609862 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:55.609963 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:55.616719 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:55.616804 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:55.616917 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:01:55.626618 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:35779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=32935
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:55.627970 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:55.629532 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:55.641184 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:55.642536 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:55.642566 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:55.643234 7466 server_base.cc:1034] running on GCE node
I20260809 15:01:55.643718 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:55.644493 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:55.645677 7466 hybrid_clock.cc:648] HybridClock initialized: now 1786287715645638 us; error 32 us; skew 500 ppm
I20260809 15:01:55.645978 7466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:55.648849 7466 init.cc:377] Logged in from keytab as oryx/127.27.186.130@KRBTEST.COM (short username oryx)
I20260809 15:01:55.650219 7466 webserver.cc:460] Webserver started at http://127.27.186.130:32935/ using document root <none> and password file <none>
I20260809 15:01:55.650841 7466 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:55.650967 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:55.655822 7466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 15:01:55.658344 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:55.659477 7466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 15:01:55.659622 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "065727ab9d7f4b61bda3adb6a82a66ac"
format_stamp: "Formatted at 2026-08-09 15:01:52 on dist-test-slave-q9h9"
server_key: "931ffd99a6d180e27a02a2b8c6980911"
server_key_iv: "9366a029e1e60e1562db7e9e8c117519"
server_key_version: "encryptionkey@0"
I20260809 15:01:55.660507 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:55.675179 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:55.678926 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:55.679196 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:55.680020 7466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:55.681217 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:55.681311 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:55.681411 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:55.681478 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:55.713724 7466 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:35779
I20260809 15:01:55.713791 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:35779 every 8 connection(s)
I20260809 15:01:55.715093 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:01:55.717296 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 7466
I20260809 15:01:55.717585 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 6397
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.131@KRBTEST.COM
Aug 09 15:01:55 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287715, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.188@KRBTEST.COM
I20260809 15:01:55.765893 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.718511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.27.186.130:56089)
0809 15:01:55.739210 (+ 20699us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.739228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.739340 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.746599 (+ 7259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.746604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.747199 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:55.747498 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.747509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.749039 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.749046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.750970 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.750982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.752691 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.752699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.752981 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.754709 (+ 1728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:55.754732 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:55.758436 (+ 3704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:55.762871 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.762876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.762879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.763473 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.763911 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.763916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.763919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.764072 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.764822 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:55.764827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:55.764969 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.765124 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.765366 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20560,"mutex_wait_us":236,"thread_start_us":27860,"threads_started":1}
I20260809 15:01:55.766317 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.718359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.27.186.130:39811)
0809 15:01:55.745563 (+ 27204us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.745582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.745689 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.751610 (+ 5921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.751615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.752140 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:55.752414 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.752425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.753778 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.753783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.755341 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.755355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.756296 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.756304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.756480 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.757014 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:55.757036 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:55.759497 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:55.763108 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.763114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.763118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.763706 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.764134 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.764139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.764142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.764300 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.765072 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:55.765077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:55.765214 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.765368 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.765621 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27046,"mutex_wait_us":4315,"thread_start_us":16192,"threads_started":1}
I20260809 15:01:55.766573 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.719128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:54887 (local address 127.27.186.190:36429)
0809 15:01:55.741019 (+ 21891us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.741026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.741044 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:01:55.741118 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.741123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.741199 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.741457 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.743756 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.745040 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.750900 (+ 5860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.751547 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.751675 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.757650 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.757655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.757658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.757689 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.761051 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.761931 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.761937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.761940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.762152 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.762680 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.762686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.762734 (+ 48us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.763086 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.763240 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.765973 (+ 2733us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.766232 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21758,"thread_start_us":21662,"threads_started":1}
I20260809 15:01:55.767064 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.718359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:54887)
0809 15:01:55.733342 (+ 14983us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:55.733393 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:55.735257 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:55.741743 (+ 6486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:55.741755 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:55.742558 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:55.743338 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.743363 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.748519 (+ 5156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.748526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.750715 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:55.750730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.752917 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.752924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:55.753110 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.754099 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:55.754134 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:55.757414 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:55.761225 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.761234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.761249 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.761787 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.762292 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:55.762298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:55.762302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:55.762540 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:55.763331 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:55.763338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:55.763724 (+ 386us) client_negotiation.cc:769] Sending connection context
0809 15:01:55.764116 (+ 392us) client_negotiation.cc:240] Negotiation successful
0809 15:01:55.764584 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14689,"mutex_wait_us":1,"thread_start_us":14456,"threads_started":1}
I20260809 15:01:55.767099 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.718829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56089 (local address 127.27.186.189:45513)
0809 15:01:55.746129 (+ 27300us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.746136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.746155 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:55.746216 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.746221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.746299 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.746404 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.747689 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.748853 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.751895 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.752491 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.752614 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.758650 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.758657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.758661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.758699 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.762736 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.763622 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.763627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.763630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.763780 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.764188 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.764193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.764196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.764597 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.764763 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.766551 (+ 1788us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.766768 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27146,"thread_start_us":27043,"threads_started":1}
I20260809 15:01:55.767248 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.719706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.27.186.131:35347)
0809 15:01:55.739895 (+ 20189us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.739905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.739939 (+ 34us) server_negotiation.cc:407] Connection header received
0809 15:01:55.740025 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.740034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.740265 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.740487 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.743736 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.745397 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.752205 (+ 6808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.752982 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.753105 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.759731 (+ 6626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.759743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.759757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.759797 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.763345 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.764118 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.764126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.764128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.764302 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.764768 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.764773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.764776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.765161 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.765444 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.766554 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.766800 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19995,"thread_start_us":19846,"threads_started":1}
I20260809 15:01:55.767581 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:55.717771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:39811 (local address 127.27.186.188:40181)
0809 15:01:55.750932 (+ 33161us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:55.750938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:55.750963 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:01:55.751018 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:55.751023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:55.751088 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:55.751405 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:55.752581 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.753606 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.755516 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:55.756089 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:55.756225 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:55.759689 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:55.759695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:55.759699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:55.759736 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:55.762949 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.763866 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.763871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.763874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.764010 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:55.764432 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:55.764437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:55.764440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:55.764860 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:55.765003 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:55.767050 (+ 2047us) server_negotiation.cc:299] Negotiation successful
0809 15:01:55.767255 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33048,"thread_start_us":32948,"threads_started":1}
I20260809 15:01:55.768317 7689 heartbeater.cc:347] Connected to a master server at 127.27.186.189:45513
I20260809 15:01:55.768456 7683 heartbeater.cc:347] Connected to a master server at 127.27.186.188:40181
I20260809 15:01:55.768571 6588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:55.768630 7683 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:55.768635 7689 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:55.769550 7689 heartbeater.cc:511] Master 127.27.186.189:45513 requested a full tablet report, sending...
I20260809 15:01:55.770304 7686 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36429
I20260809 15:01:55.770430 7686 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:55.770841 7686 heartbeater.cc:511] Master 127.27.186.190:36429 requested a full tablet report, sending...
I20260809 15:01:55.771204 7683 heartbeater.cc:511] Master 127.27.186.188:40181 requested a full tablet report, sending...
I20260809 15:01:55.771337 6841 ts_manager.cc:194] Registered new tserver with Master: 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779)
I20260809 15:01:55.772111 7147 ts_manager.cc:194] Registered new tserver with Master: 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779)
I20260809 15:01:55.772181 5855 ts_manager.cc:194] Re-registered known tserver with Master: 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779)
I20260809 15:01:55.773324 5855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:54887
I20260809 15:01:55.853389 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 6397
I20260809 15:01:55.871840 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:35347
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:36429
--webserver_port=34313
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:01:56.181528 7438 heartbeater.cc:503] Master 127.27.186.190:36429 was elected leader, sending a full tablet report...
W20260809 15:01:56.184535 7702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:01:56.184782 7702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:01:56.184921 7702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:01:56.192016 7702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:01:56.192124 7702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:01:56.192267 7702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:01:56.201571 7702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:35347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=34313
--tserver_master_addrs=127.27.186.190:36429,127.27.186.189:45513,127.27.186.188:40181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:01:56.202735 7702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:01:56.203943 7702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:01:56.216337 7711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:56.216305 7713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:01:56.216420 7717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:01:56.217314 7702 server_base.cc:1034] running on GCE node
I20260809 15:01:56.217891 7702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:01:56.218859 7702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:01:56.220000 7702 hybrid_clock.cc:648] HybridClock initialized: now 1786287716219950 us; error 45 us; skew 500 ppm
I20260809 15:01:56.220304 7702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:01:56 dist-test-slave-q9h9 krb5kdc[5790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287716, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:01:56.223217 7702 init.cc:377] Logged in from keytab as oryx/127.27.186.131@KRBTEST.COM (short username oryx)
I20260809 15:01:56.224568 7702 webserver.cc:460] Webserver started at http://127.27.186.131:34313/ using document root <none> and password file <none>
I20260809 15:01:56.225216 7702 fs_manager.cc:359] Metadata directory not provided
I20260809 15:01:56.225346 7702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:01:56.230456 7702 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 15:01:56.233264 7730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:56.234665 7702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:01:56.234840 7702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "c021a3a60d38450381b12668e655c4df"
format_stamp: "Formatted at 2026-08-09 15:01:53 on dist-test-slave-q9h9"
server_key: "ce8dbf18b19ded4a618f0f18cef181d1"
server_key_iv: "2ccf77c9c95050b73aedd92cfcecabb1"
server_key_version: "encryptionkey@0"
I20260809 15:01:56.235828 7702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:01:56.254446 7702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:01:56.257990 7702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:01:56.258246 7702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:01:56.259104 7702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:01:56.260370 7702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:01:56.260462 7702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:56.260567 7702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:01:56.260651 7702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:01:56.289685 7702 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:35347
I20260809 15:01:56.289747 7929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:35347 every 8 connection(s)
I20260809 15:01:56.291020 7702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:01:56.291317 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 7702
Aug 09 15:01:56 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287716, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.188@KRBTEST.COM
Aug 09 15:01:56 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287716, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.189@KRBTEST.COM
Aug 09 15:01:56 dist-test-slave-q9h9 krb5kdc[5790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287716, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:01:56.338891 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.294943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.27.186.131:46297)
0809 15:01:56.313036 (+ 18093us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.313092 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.315642 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.319478 (+ 3836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.319483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.321085 (+ 1602us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:56.321263 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.321271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.322377 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.322383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.324321 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.324336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.325136 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.325145 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.325350 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.326033 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:56.326065 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:56.329156 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:56.334862 (+ 5706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:56.334867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:56.334870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:56.335350 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:56.335787 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:56.335792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:56.335796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:56.335951 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:56.336579 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:56.336584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:56.336774 (+ 190us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.336972 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.337493 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17784,"mutex_wait_us":7,"thread_start_us":21336,"threads_started":1}
I20260809 15:01:56.339740 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.310742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.27.186.131:43161)
0809 15:01:56.316679 (+ 5937us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.316696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.316815 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.319337 (+ 2522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.319342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.320298 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:56.320888 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.320914 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.324134 (+ 3220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.324141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.325789 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.325805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.327005 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.327013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.327198 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.327743 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:56.327759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:56.329882 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:56.333872 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:56.333881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:56.333895 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:56.334510 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:56.335097 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:56.335102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:56.335105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:56.335371 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:56.336107 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:56.336116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:56.336520 (+ 404us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.337511 (+ 991us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.337785 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5814,"mutex_wait_us":2,"thread_start_us":3356,"threads_started":1}
I20260809 15:01:56.340214 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.301387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:49203)
0809 15:01:56.314440 (+ 13053us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.314459 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.315646 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.318805 (+ 3159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.318815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.319950 (+ 1135us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:01:56.320888 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.320914 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.322755 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.322761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.324320 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.324336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.327397 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.327405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.328382 (+ 977us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.329662 (+ 1280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:01:56.329685 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:01:56.332380 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:01:56.336844 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:56.336850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:56.336854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:56.337348 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:56.337931 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:01:56.337936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:01:56.337941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:01:56.338119 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:01:56.338756 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:01:56.338762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:01:56.338976 (+ 214us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.339104 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.339371 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12909,"mutex_wait_us":40,"thread_start_us":11231,"threads_started":1}
I20260809 15:01:56.340509 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.293857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:46297 (local address 127.27.186.188:40181)
0809 15:01:56.319000 (+ 25143us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.319006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.319025 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:56.319082 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.319086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.319156 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.319470 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.321401 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.322226 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.324496 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.324889 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.324997 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.330158 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.330164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.330168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.330204 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.334692 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.335508 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.335513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.335517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.335656 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.336095 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.336099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.336102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.336432 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.339622 (+ 3190us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.339749 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.339949 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25023,"thread_start_us":25796,"threads_started":1}
I20260809 15:01:56.340518 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.293768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:49203 (local address 127.27.186.190:36429)
0809 15:01:56.309835 (+ 16067us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.309839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.318078 (+ 8239us) server_negotiation.cc:407] Connection header received
0809 15:01:56.318159 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.318165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.318255 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.318573 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.321424 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.322594 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.325985 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.326672 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.326810 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.332613 (+ 5803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.332621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.332626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.332668 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.336069 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.337454 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.337460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.337463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.337622 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.338213 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.338217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.338220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.338560 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.338691 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.339537 (+ 846us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.339750 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15943,"thread_start_us":16053,"threads_started":1}
I20260809 15:01:56.341260 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.311169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:43161 (local address 127.27.186.189:45513)
0809 15:01:56.319042 (+ 7873us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.319048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.319068 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:56.319124 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.319130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.319193 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.319284 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.322809 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.323940 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.326174 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.326791 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.327502 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.330092 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.330098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.330102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.330140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.333721 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.334757 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.334762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.334766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.334943 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.335549 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.335554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.335558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.335942 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.340548 (+ 4606us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.340685 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.340923 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7757,"thread_start_us":7653,"threads_started":1}
I20260809 15:01:56.341920 7931 heartbeater.cc:347] Connected to a master server at 127.27.186.188:40181
I20260809 15:01:56.342262 7931 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:56.343112 7931 heartbeater.cc:511] Master 127.27.186.188:40181 requested a full tablet report, sending...
I20260809 15:01:56.344374 7936 heartbeater.cc:347] Connected to a master server at 127.27.186.189:45513
I20260809 15:01:56.344496 7936 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:56.344954 7936 heartbeater.cc:511] Master 127.27.186.189:45513 requested a full tablet report, sending...
I20260809 15:01:56.346130 6841 ts_manager.cc:194] Registered new tserver with Master: c021a3a60d38450381b12668e655c4df (127.27.186.131:35347)
I20260809 15:01:56.352650 7147 ts_manager.cc:194] Registered new tserver with Master: c021a3a60d38450381b12668e655c4df (127.27.186.131:35347)
I20260809 15:01:56.352813 7932 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36429
I20260809 15:01:56.352948 7932 heartbeater.cc:464] Registering TS with master...
I20260809 15:01:56.353413 7932 heartbeater.cc:511] Master 127.27.186.190:36429 requested a full tablet report, sending...
I20260809 15:01:56.354403 5855 ts_manager.cc:194] Re-registered known tserver with Master: c021a3a60d38450381b12668e655c4df (127.27.186.131:35347)
I20260809 15:01:56.355448 5855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:49203
I20260809 15:01:56.368074 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.341606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.27.186.188:40181)
0809 15:01:56.344692 (+ 3086us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.344697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.351438 (+ 6741us) server_negotiation.cc:407] Connection header received
0809 15:01:56.352536 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.352541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.352650 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.352715 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.356056 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.357026 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.359594 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.360174 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.360321 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.362835 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.362841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.362845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.362880 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.365404 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.366174 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.366179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.366182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.366317 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.366659 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.366664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.366667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.367023 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.367248 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.367533 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.367740 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":10190,"threads_started":1}
I20260809 15:01:56.368311 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.336318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.27.186.190:36429)
0809 15:01:56.341043 (+ 4725us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.341049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.349736 (+ 8687us) server_negotiation.cc:407] Connection header received
0809 15:01:56.349903 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.349909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.349982 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.350093 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.354141 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.355206 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.357959 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.358581 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.358704 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.362524 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.362531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.362536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.362570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.365653 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.366453 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.366458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.366462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.366567 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.366872 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.366877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.366880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.367263 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.367486 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.367801 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.368002 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4635,"thread_start_us":15381,"threads_started":1}
I20260809 15:01:56.370693 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.348898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.27.186.189:45513)
0809 15:01:56.349056 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.349061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.352718 (+ 3657us) server_negotiation.cc:407] Connection header received
0809 15:01:56.352895 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.352900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.352982 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.353049 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.354961 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.355957 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.357945 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.358593 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.358704 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.362895 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.362899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.362902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.362929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.365918 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.368614 (+ 2696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.368618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.368620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.368755 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.369148 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.369153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.369156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.369468 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.369702 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.370202 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.370415 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:01:56.371984 5855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:56.378540 5855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:56.417830 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.394577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.27.186.131:35347)
0809 15:01:56.407894 (+ 13317us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.407903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.407937 (+ 34us) server_negotiation.cc:407] Connection header received
0809 15:01:56.408037 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.408045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.408527 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.408718 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:56.409760 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.411774 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.414710 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.416141 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.416305 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.416477 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.417050 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.417525 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13121,"thread_start_us":13738,"threads_started":1}
I20260809 15:01:56.418045 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.394327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:35347 (local address 127.0.0.1:56716)
0809 15:01:56.405648 (+ 11321us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.405668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.405846 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.409014 (+ 3168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.409020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.409047 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:56.409543 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.409556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.411928 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.411934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.414311 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.414326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.416582 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.416589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.416758 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.416795 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.416907 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.417121 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11193,"mutex_wait_us":40,"thread_start_us":19697,"threads_started":1}
I20260809 15:01:56.421969 7823 tablet_service.cc:1467] Processing CreateTablet for tablet 2556f975c98c4cb6970765b3264bdda4 (DEFAULT_TABLE table=test-table [id=0ed3d3a872db4412aa6308c5e06363c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:56.423478 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556f975c98c4cb6970765b3264bdda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:56.424309 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.394891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:36657 (local address 127.0.0.1:37722)
0809 15:01:56.414406 (+ 19515us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.414425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.414554 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.415736 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.415742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.415771 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:56.416104 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.416116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.418414 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.418420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.421350 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.421365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.423402 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.423408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.423562 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.423594 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.423807 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.423937 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19403,"mutex_wait_us":517,"thread_start_us":10498,"threads_started":1}
I20260809 15:01:56.424561 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.395551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.27.186.129:36657)
0809 15:01:56.414969 (+ 19418us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.414977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.415006 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:01:56.415103 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.415111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.415466 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.415691 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:56.416310 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.417744 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.421535 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.422961 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.423103 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.423272 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.423852 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.424028 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19237,"thread_start_us":19130,"threads_started":1}
I20260809 15:01:56.425859 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.394567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:35779 (local address 127.0.0.1:45832)
0809 15:01:56.414009 (+ 19442us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.414043 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.414173 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.416733 (+ 2560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.416737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.416761 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:56.419037 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.419052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.420817 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.420823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.422871 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.422886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.425025 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.425033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.425242 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.425274 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.425358 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.425471 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19315,"mutex_wait_us":21,"thread_start_us":19158,"threads_started":1}
I20260809 15:01:56.425935 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.395123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.27.186.130:35779)
0809 15:01:56.416304 (+ 21181us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.416309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.416331 (+ 22us) server_negotiation.cc:407] Connection header received
0809 15:01:56.416403 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.416408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.416553 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.416715 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:56.419294 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.420694 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.423060 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.424469 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.424626 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.424805 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.425496 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.425673 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21027,"thread_start_us":20924,"threads_started":1}
I20260809 15:01:56.427771 7322 tablet_service.cc:1467] Processing CreateTablet for tablet 2556f975c98c4cb6970765b3264bdda4 (DEFAULT_TABLE table=test-table [id=0ed3d3a872db4412aa6308c5e06363c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:56.428836 7322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556f975c98c4cb6970765b3264bdda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:56.436275 8001 tablet_bootstrap.cc:492] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Bootstrap starting.
I20260809 15:01:56.436658 7570 tablet_service.cc:1467] Processing CreateTablet for tablet 2556f975c98c4cb6970765b3264bdda4 (DEFAULT_TABLE table=test-table [id=0ed3d3a872db4412aa6308c5e06363c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:56.438076 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556f975c98c4cb6970765b3264bdda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:56.439766 8001 tablet_bootstrap.cc:654] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:56.440987 8001 log.cc:824] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:56.443471 8001 tablet_bootstrap.cc:492] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: No bootstrap required, opened a new log
I20260809 15:01:56.443717 8001 ts_tablet_manager.cc:1397] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 15:01:56.446581 8006 tablet_bootstrap.cc:492] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Bootstrap starting.
I20260809 15:01:56.450197 8006 tablet_bootstrap.cc:654] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:56.450500 8008 tablet_bootstrap.cc:492] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Bootstrap starting.
I20260809 15:01:56.451537 8006 log.cc:824] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:56.452677 8001 raft_consensus.cc:348] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.453025 8001 raft_consensus.cc:374] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:56.453125 8001 raft_consensus.cc:729] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c021a3a60d38450381b12668e655c4df, State: Initialized, Role: FOLLOWER
I20260809 15:01:56.453704 8008 tablet_bootstrap.cc:654] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:56.453743 8001 consensus_queue.cc:260] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.454859 8008 log.cc:824] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Log is configured to *not* fsync() on all Append() calls
I20260809 15:01:56.455437 8001 ts_tablet_manager.cc:1428] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 15:01:56.455796 7932 heartbeater.cc:503] Master 127.27.186.190:36429 was elected leader, sending a full tablet report...
I20260809 15:01:56.456259 8006 tablet_bootstrap.cc:492] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: No bootstrap required, opened a new log
I20260809 15:01:56.456523 8006 ts_tablet_manager.cc:1397] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 15:01:56.460938 8008 tablet_bootstrap.cc:492] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: No bootstrap required, opened a new log
I20260809 15:01:56.461208 8008 ts_tablet_manager.cc:1397] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260809 15:01:56.466508 8006 raft_consensus.cc:348] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.466909 8006 raft_consensus.cc:374] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:56.467011 8006 raft_consensus.cc:729] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9d3ce19fa54354a136d05df2e352d0, State: Initialized, Role: FOLLOWER
I20260809 15:01:56.467561 8006 consensus_queue.cc:260] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.469082 8006 ts_tablet_manager.cc:1428] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 15:01:56.471027 8008 raft_consensus.cc:348] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.471431 8008 raft_consensus.cc:374] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:56.471524 8008 raft_consensus.cc:729] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065727ab9d7f4b61bda3adb6a82a66ac, State: Initialized, Role: FOLLOWER
I20260809 15:01:56.472095 8008 consensus_queue.cc:260] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.473701 8008 ts_tablet_manager.cc:1428] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260809 15:01:56.473913 7686 heartbeater.cc:503] Master 127.27.186.190:36429 was elected leader, sending a full tablet report...
I20260809 15:01:56.496780 8015 raft_consensus.cc:482] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:56.496970 8015 raft_consensus.cc:504] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.498379 8015 leader_election.cc:288] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c021a3a60d38450381b12668e655c4df (127.27.186.131:35347), cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657)
I20260809 15:01:56.504694 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.498790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:60395 (local address 127.27.186.129:36657)
0809 15:01:56.498970 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.498976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.499714 (+ 738us) server_negotiation.cc:407] Connection header received
0809 15:01:56.499863 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.499868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.499935 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.500027 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:56.500591 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.501295 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.503057 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.503806 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.503949 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.504018 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.504367 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.504485 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:01:56.504688 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.498598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56125 (local address 127.27.186.131:35347)
0809 15:01:56.498796 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.498803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.499525 (+ 722us) server_negotiation.cc:407] Connection header received
0809 15:01:56.499684 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.499689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.499771 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.499849 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:56.500389 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.501149 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.503064 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.503791 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.503921 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.503994 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.504342 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.504464 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:01:56.505038 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.498521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:35347 (local address 127.27.186.130:56125)
0809 15:01:56.499375 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.499389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.499551 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.499866 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.499871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.499895 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:56.500240 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.500255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.501280 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.501286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.502907 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.502920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.503999 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.504006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.504176 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.504217 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.504290 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.504403 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":93,"thread_start_us":763,"threads_started":1}
I20260809 15:01:56.505065 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.498592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:36657 (local address 127.27.186.130:60395)
0809 15:01:56.499549 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:56.499570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:56.499753 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:56.500056 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:56.500059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:56.500078 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:56.500417 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.500430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.501430 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.501434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.502920 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:56.502932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.504056 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.504061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:56.504221 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.504243 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 15:01:56.504329 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 15:01:56.504457 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":17,"thread_start_us":421,"threads_started":1}
I20260809 15:01:56.505864 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556f975c98c4cb6970765b3264bdda4" candidate_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9d3ce19fa54354a136d05df2e352d0" is_pre_election: true
I20260809 15:01:56.505867 7859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556f975c98c4cb6970765b3264bdda4" candidate_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c021a3a60d38450381b12668e655c4df" is_pre_election: true
I20260809 15:01:56.506271 7351 raft_consensus.cc:2454] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065727ab9d7f4b61bda3adb6a82a66ac in term 0.
I20260809 15:01:56.506304 7859 raft_consensus.cc:2454] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065727ab9d7f4b61bda3adb6a82a66ac in term 0.
I20260809 15:01:56.506815 7496 leader_election.cc:302] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065727ab9d7f4b61bda3adb6a82a66ac, cc9d3ce19fa54354a136d05df2e352d0; no voters:
I20260809 15:01:56.507157 8015 raft_consensus.cc:2781] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:56.507310 8015 raft_consensus.cc:482] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:56.507401 8015 raft_consensus.cc:3037] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:56.509791 8015 raft_consensus.cc:504] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.510416 8015 leader_election.cc:288] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [CANDIDATE]: Term 1 election: Requested vote from peers c021a3a60d38450381b12668e655c4df (127.27.186.131:35347), cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657)
I20260809 15:01:56.510695 7859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556f975c98c4cb6970765b3264bdda4" candidate_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c021a3a60d38450381b12668e655c4df"
I20260809 15:01:56.510860 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556f975c98c4cb6970765b3264bdda4" candidate_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9d3ce19fa54354a136d05df2e352d0"
I20260809 15:01:56.510892 7859 raft_consensus.cc:3037] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:56.511047 7351 raft_consensus.cc:3037] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:56.514053 7859 raft_consensus.cc:2454] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 065727ab9d7f4b61bda3adb6a82a66ac in term 1.
I20260809 15:01:56.514053 7351 raft_consensus.cc:2454] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 065727ab9d7f4b61bda3adb6a82a66ac in term 1.
I20260809 15:01:56.514379 7496 leader_election.cc:302] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065727ab9d7f4b61bda3adb6a82a66ac, cc9d3ce19fa54354a136d05df2e352d0; no voters:
I20260809 15:01:56.514626 8015 raft_consensus.cc:2781] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:56.515002 8015 raft_consensus.cc:686] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 LEADER]: Becoming Leader. State: Replica: 065727ab9d7f4b61bda3adb6a82a66ac, State: Running, Role: LEADER
I20260809 15:01:56.515411 8015 consensus_queue.cc:237] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } }
I20260809 15:01:56.519974 5855 catalog_manager.cc:5495] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130). New cstate: current_term: 1 leader_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } health_report { overall_health: UNKNOWN } } }
I20260809 15:01:56.543735 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.538822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.27.186.130:35779)
0809 15:01:56.538986 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.538993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.539118 (+ 125us) server_negotiation.cc:407] Connection header received
0809 15:01:56.539340 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.539344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.539419 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.539536 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:56.539890 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.540637 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.541868 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.542298 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.542387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.542813 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:56.543282 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:56.543384 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.543491 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.543551 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 15:01:56.550828 7351 raft_consensus.cc:1261] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Refusing update from remote peer 065727ab9d7f4b61bda3adb6a82a66ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:56.550828 7859 raft_consensus.cc:1261] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Refusing update from remote peer 065727ab9d7f4b61bda3adb6a82a66ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:56.551640 8015 consensus_queue.cc:1035] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:56.551894 8019 consensus_queue.cc:1035] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:01:56.559091 7940 tablet.cc:2369] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:56.577432 8026 mvcc.cc:205] Tried to move back new op lower bound from 7316634486984294400 to 7316634486850609152. Current Snapshot: MvccSnapshot[applied={T|T < 7316634486984294400}]
I20260809 15:01:56.579420 5855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:56.579705 5855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:56.586115 5855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8df908b68a034b08b5862e491b99c8a9: Sending DeleteTablet for 3 replicas of tablet 2556f975c98c4cb6970765b3264bdda4
I20260809 15:01:56.587556 7322 tablet_service.cc:1514] Processing DeleteTablet for tablet 2556f975c98c4cb6970765b3264bdda4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:56 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:37722
I20260809 15:01:56.587657 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet 2556f975c98c4cb6970765b3264bdda4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:56 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:56716
I20260809 15:01:56.587945 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet 2556f975c98c4cb6970765b3264bdda4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:56 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45832
I20260809 15:01:56.600643 8039 tablet_replica.cc:330] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: stopping tablet replica
I20260809 15:01:56.601074 8039 raft_consensus.cc:2229] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:56.601570 8039 raft_consensus.cc:2258] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:56.601755 8040 tablet_replica.cc:330] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: stopping tablet replica
I20260809 15:01:56.602231 8040 raft_consensus.cc:2229] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:56.602864 8040 raft_consensus.cc:2258] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:56.603008 8038 tablet_replica.cc:330] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: stopping tablet replica
I20260809 15:01:56.603834 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 5808
I20260809 15:01:56.604848 5901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:56.604895 8040 ts_tablet_manager.cc:1905] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:56.603369 8038 raft_consensus.cc:2229] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:56.607193 8038 raft_consensus.cc:2258] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:56.609431 8038 ts_tablet_manager.cc:1905] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:56.611069 8039 ts_tablet_manager.cc:1905] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:56.613272 8040 ts_tablet_manager.cc:1918] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:56.613432 8040 log.cc:1196] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/2556f975c98c4cb6970765b3264bdda4
I20260809 15:01:56.614065 8040 ts_tablet_manager.cc:1939] T 2556f975c98c4cb6970765b3264bdda4 P 065727ab9d7f4b61bda3adb6a82a66ac: Deleting consensus metadata
I20260809 15:01:56.624150 8038 ts_tablet_manager.cc:1918] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:56.624287 8038 log.cc:1196] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/2556f975c98c4cb6970765b3264bdda4
I20260809 15:01:56.624694 8038 ts_tablet_manager.cc:1939] T 2556f975c98c4cb6970765b3264bdda4 P cc9d3ce19fa54354a136d05df2e352d0: Deleting consensus metadata
I20260809 15:01:56.630541 8039 ts_tablet_manager.cc:1918] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:56.630718 8039 log.cc:1196] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/2556f975c98c4cb6970765b3264bdda4
I20260809 15:01:56.631276 8039 ts_tablet_manager.cc:1939] T 2556f975c98c4cb6970765b3264bdda4 P c021a3a60d38450381b12668e655c4df: Deleting consensus metadata
I20260809 15:01:56.635730 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.589157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.27.186.189:45513)
0809 15:01:56.589309 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.589316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.602535 (+ 13219us) server_negotiation.cc:407] Connection header received
0809 15:01:56.604221 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.604226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.604304 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.604384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.614140 (+ 9756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.615328 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.617331 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.618049 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.618179 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.629458 (+ 11279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.629465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.629470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.629517 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.633062 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.633738 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.633744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.633748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.633884 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.634314 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.634320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.634324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.634601 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.634893 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.635187 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.635388 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:01:56.647940 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.602620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.27.186.188:40181)
0809 15:01:56.602807 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.602813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.635916 (+ 33103us) server_negotiation.cc:407] Connection header received
0809 15:01:56.636308 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.636313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.636411 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.636480 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.638118 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.638948 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.640369 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.640765 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.640880 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.643166 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.643171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.643174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.643205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.645537 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.646138 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.646142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.646144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.646269 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.646609 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.646612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.646614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.646884 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.647107 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.647415 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.647608 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 15:01:56.785239 5831 catalog_manager.cc:4841] TS cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657): tablet 2556f975c98c4cb6970765b3264bdda4 (table test-table [id=0ed3d3a872db4412aa6308c5e06363c1]) successfully deleted
I20260809 15:01:56.785622 5834 catalog_manager.cc:4841] TS 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779): tablet 2556f975c98c4cb6970765b3264bdda4 (table test-table [id=0ed3d3a872db4412aa6308c5e06363c1]) successfully deleted
W20260809 15:01:56.785858 5834 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.186.130:35779, user_credentials={real_user=slave}} blocked reactor thread for 170124us
I20260809 15:01:56.787514 5837 catalog_manager.cc:4841] TS c021a3a60d38450381b12668e655c4df (127.27.186.131:35347): tablet 2556f975c98c4cb6970765b3264bdda4 (table test-table [id=0ed3d3a872db4412aa6308c5e06363c1]) successfully deleted
I20260809 15:01:56.791635 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 5808
I20260809 15:01:56.856734 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.831299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.27.186.188:40181)
0809 15:01:56.831524 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.831530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.839125 (+ 7595us) server_negotiation.cc:407] Connection header received
0809 15:01:56.839616 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.839622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.839710 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.839788 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.842849 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.843992 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.846008 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.846666 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.846759 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.850362 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.850369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.850373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.850439 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.853843 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.854607 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.854612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.854616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.854762 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.855168 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.855173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.855176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.855559 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.855808 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.856167 (+ 359us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.856400 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:01:56.858522 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:56.831304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.27.186.189:45513)
0809 15:01:56.831577 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:56.831584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:56.843958 (+ 12374us) server_negotiation.cc:407] Connection header received
0809 15:01:56.844441 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:56.844448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:56.844531 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:56.844600 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:56.847095 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.848325 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.849621 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:56.850100 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:56.850208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:56.852805 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:56.852810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:56.852813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:56.852845 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:56.855500 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.856239 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.856244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.856247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.856516 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:56.856974 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:56.856980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:56.856983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:56.857327 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:56.857580 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:56.857951 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 15:01:56.858191 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 15:01:57.571878 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:57.571001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:44909)
0809 15:01:57.571498 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:57.571656 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":267,"threads_started":1}
W20260809 15:01:57.572342 7932 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:36429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:57.579243 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:57.578610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:51281)
0809 15:01:57.578972 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:57.579079 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":186,"threads_started":1}
W20260809 15:01:57.579559 7686 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:36429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:57.591117 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:57.590477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:53279)
0809 15:01:57.590839 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:57.590945 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":169,"threads_started":1}
W20260809 15:01:57.591476 7438 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:36429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:58.087134 8060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8df908b68a034b08b5862e491b99c8a9)
I20260809 15:01:58.087374 8060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } }
I20260809 15:01:58.088819 8060 leader_election.cc:288] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429), e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189:45513)
I20260809 15:01:58.089939 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.088900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37774)
0809 15:01:58.089421 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.089624 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"spinlock_wait_cycles":4352,"thread_start_us":407,"threads_started":1}
W20260809 15:01:58.090373 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:58.090975 8067 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8df908b68a034b08b5862e491b99c8a9)
I20260809 15:01:58.091346 8067 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } }
I20260809 15:01:58.099071 8067 leader_election.cc:288] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429), 525c52b6bdc2459db16a822408954503 (127.27.186.188:40181)
I20260809 15:01:58.099471 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.092938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37780)
0809 15:01:58.098947 (+ 6009us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.099169 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5893,"thread_start_us":5738,"threads_started":1}
W20260809 15:01:58.100371 6812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:58.107116 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.088991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:45513 (local address 127.0.0.1:41348)
0809 15:01:58.090300 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.090316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.090463 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.100140 (+ 9677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.100146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.100197 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.100460 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.100472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.102436 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.102442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.104532 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.104548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.106315 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.106322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.106516 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.106557 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.106650 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.106805 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"threads_started":1}
I20260809 15:01:58.107537 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.104434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37788)
0809 15:01:58.107305 (+ 2871us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.107391 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2783}
W20260809 15:01:58.107668 7127 leader_election.cc:334] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429): Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:58.108531 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.107613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37802)
0809 15:01:58.108092 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.108285 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260809 15:01:58.108675 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.089069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.27.186.189:45513)
0809 15:01:58.099800 (+ 10731us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.099807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.099827 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:58.099894 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.099900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.099972 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.100644 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.100863 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.102250 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.104734 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.106107 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.107983 (+ 1876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.108054 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.108110 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.108269 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10601,"thread_start_us":10332,"threads_started":1}
W20260809 15:01:58.108703 6812 leader_election.cc:334] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429): Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:58.109195 6853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "525c52b6bdc2459db16a822408954503" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e32d364ca0db419a8d52ccc495486ec6" is_pre_election: true
I20260809 15:01:58.109437 6853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525c52b6bdc2459db16a822408954503 in term 1.
I20260809 15:01:58.109851 7131 leader_election.cc:302] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525c52b6bdc2459db16a822408954503, e32d364ca0db419a8d52ccc495486ec6; no voters: 8df908b68a034b08b5862e491b99c8a9
I20260809 15:01:58.110200 8060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:01:58.110321 8060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8df908b68a034b08b5862e491b99c8a9)
I20260809 15:01:58.110407 8060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:01:58.113034 8060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } }
I20260809 15:01:58.113701 8060 leader_election.cc:288] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [CANDIDATE]: Term 2 election: Requested vote from peers 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429), e32d364ca0db419a8d52ccc495486ec6 (127.27.186.189:45513)
I20260809 15:01:58.114244 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.099289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:40181 (local address 127.0.0.1:43390)
0809 15:01:58.107367 (+ 8078us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.107388 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.107508 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.108068 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.108073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.108102 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.108421 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.108433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.109887 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.109893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.112284 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.112294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.113522 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.113528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.113689 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.113731 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.113826 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.113954 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7966,"thread_start_us":8564,"threads_started":1}
I20260809 15:01:58.114337 6853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "525c52b6bdc2459db16a822408954503" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e32d364ca0db419a8d52ccc495486ec6"
I20260809 15:01:58.114450 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.099739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.27.186.188:40181)
0809 15:01:58.107032 (+ 7293us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.107038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.107611 (+ 573us) server_negotiation.cc:407] Connection header received
0809 15:01:58.107674 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.107678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.107757 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.107854 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.108616 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.109778 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.112422 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.113343 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.113530 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.113597 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.113967 (+ 370us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.114112 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7150,"thread_start_us":7026,"threads_started":1}
I20260809 15:01:58.114531 6853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:01:58.114756 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.113808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37828)
0809 15:01:58.114369 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.114540 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260809 15:01:58.115294 7167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e32d364ca0db419a8d52ccc495486ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "525c52b6bdc2459db16a822408954503" is_pre_election: true
I20260809 15:01:58.115644 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.115323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37836)
0809 15:01:58.115459 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.115537 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 15:01:58.115751 7167 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e32d364ca0db419a8d52ccc495486ec6 in current term 2: Already voted for candidate 525c52b6bdc2459db16a822408954503 in this term.
W20260809 15:01:58.115845 7127 leader_election.cc:334] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429): Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
I20260809 15:01:58.116358 6814 leader_election.cc:302] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e32d364ca0db419a8d52ccc495486ec6; no voters: 525c52b6bdc2459db16a822408954503, 8df908b68a034b08b5862e491b99c8a9
I20260809 15:01:58.117050 6853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525c52b6bdc2459db16a822408954503 in term 2.
I20260809 15:01:58.117228 8067 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:01:58.117417 7131 leader_election.cc:302] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525c52b6bdc2459db16a822408954503, e32d364ca0db419a8d52ccc495486ec6; no voters: 8df908b68a034b08b5862e491b99c8a9
I20260809 15:01:58.117666 8060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:01:58.118060 8060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [term 2 LEADER]: Becoming Leader. State: Replica: 525c52b6bdc2459db16a822408954503, State: Running, Role: LEADER
I20260809 15:01:58.118384 8060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } }
I20260809 15:01:58.120028 8077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525c52b6bdc2459db16a822408954503. Latest consensus state: current_term: 2 leader_uuid: "525c52b6bdc2459db16a822408954503" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:58.120416 8077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:58.121048 8081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:01:58.124873 8081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:01:58.125705 8081 catalog_manager.cc:1259] Loaded cluster ID: 8450f063ff444687a284745672ea6993
I20260809 15:01:58.125782 8081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:01:58.126958 8081 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:01:58.127777 8081 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Loaded TSK: 0
I20260809 15:01:58.128553 8081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:01:58.169950 7147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:01:58.176614 7147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:01:58.180203 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.179871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37854)
0809 15:01:58.180010 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.180076 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 15:01:58.180784 6853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [term 2 FOLLOWER]: Refusing update from remote peer 525c52b6bdc2459db16a822408954503: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 15:01:58.181012 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.180699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37860)
0809 15:01:58.180834 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.180907 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:01:58.181221 7127 consensus_peers.cc:487] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 -> Peer 8df908b68a034b08b5862e491b99c8a9 (127.27.186.190:36429): Couldn't send request to peer 8df908b68a034b08b5862e491b99c8a9. Status: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:01:58.181473 8077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 15:01:58.183748 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525c52b6bdc2459db16a822408954503. Latest consensus state: current_term: 2 leader_uuid: "525c52b6bdc2459db16a822408954503" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:58.183991 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:58.185371 8077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "525c52b6bdc2459db16a822408954503" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:58.185616 8077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503 [sys.catalog]: This master's current role is: LEADER
I20260809 15:01:58.190498 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "525c52b6bdc2459db16a822408954503" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8df908b68a034b08b5862e491b99c8a9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36429 } } peers { permanent_uuid: "e32d364ca0db419a8d52ccc495486ec6" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 45513 } attrs { promote: false } } peers { permanent_uuid: "525c52b6bdc2459db16a822408954503" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 40181 } attrs { promote: false } } }
I20260809 15:01:58.190740 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d364ca0db419a8d52ccc495486ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:01:58.203140 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.194656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:36657 (local address 127.0.0.1:35860)
0809 15:01:58.194773 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.194792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.194914 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.196520 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.196526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.196554 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.196849 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.196861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.198531 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.198537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.200603 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.200619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.202339 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.202346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.202558 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.202602 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.202671 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.202803 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 15:01:58.204351 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.194926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.27.186.129:36657)
0809 15:01:58.196151 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.196158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.196178 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:01:58.196255 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.196261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.196361 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.196998 (+ 637us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.197188 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.198354 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.200823 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.202094 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.203757 (+ 1663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.203843 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.203897 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.204033 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":983,"threads_started":1}
I20260809 15:01:58.205021 7322 tablet_service.cc:1467] Processing CreateTablet for tablet f0ffd43c491745cb83b055770b35b6e5 (DEFAULT_TABLE table=test-table [id=068d474e67e94b57b28f666766263cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:58.205514 7322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ffd43c491745cb83b055770b35b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:58.212395 8104 tablet_bootstrap.cc:492] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Bootstrap starting.
I20260809 15:01:58.212994 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.195293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:35779 (local address 127.0.0.1:50438)
0809 15:01:58.203340 (+ 8047us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.203358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.203456 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.208623 (+ 5167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.208628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.208651 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.208791 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.208799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.209684 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.209689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.211050 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.211063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.212335 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.212341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.212525 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.212560 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.212629 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.212742 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7961,"thread_start_us":11919,"threads_started":1}
I20260809 15:01:58.213929 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.195430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.27.186.130:35779)
0809 15:01:58.208371 (+ 12941us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.208375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.208390 (+ 15us) server_negotiation.cc:407] Connection header received
0809 15:01:58.208444 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.208448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.208500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.208589 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.208923 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.209559 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.211235 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.212139 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.213392 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.213465 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.213516 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.213648 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12832,"thread_start_us":12754,"threads_started":1}
I20260809 15:01:58.214202 8104 tablet_bootstrap.cc:654] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:58.215037 7570 tablet_service.cc:1467] Processing CreateTablet for tablet f0ffd43c491745cb83b055770b35b6e5 (DEFAULT_TABLE table=test-table [id=068d474e67e94b57b28f666766263cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:58.215474 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ffd43c491745cb83b055770b35b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:58.216435 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.194975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:35347 (local address 127.0.0.1:35442)
0809 15:01:58.195468 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.195486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.195583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.211377 (+ 15794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.211382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.211407 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.211607 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.211619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.213098 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.213103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.214395 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.214408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.215784 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.215789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.215965 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.215997 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.216079 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.216203 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260809 15:01:58.216578 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.195312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.27.186.131:35347)
0809 15:01:58.210978 (+ 15666us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.210984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.211003 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:58.211069 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.211074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.211155 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.211751 (+ 596us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.211963 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.212959 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.214552 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.215603 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.215801 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.215870 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.216150 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.216281 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15541,"thread_start_us":15912,"threads_started":1}
I20260809 15:01:58.217283 7823 tablet_service.cc:1467] Processing CreateTablet for tablet f0ffd43c491745cb83b055770b35b6e5 (DEFAULT_TABLE table=test-table [id=068d474e67e94b57b28f666766263cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:01:58.217592 8104 tablet_bootstrap.cc:492] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: No bootstrap required, opened a new log
I20260809 15:01:58.217694 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ffd43c491745cb83b055770b35b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:01:58.217717 8104 ts_tablet_manager.cc:1397] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 15:01:58.218490 8104 raft_consensus.cc:348] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.218704 8104 raft_consensus.cc:374] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:58.218801 8104 raft_consensus.cc:729] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9d3ce19fa54354a136d05df2e352d0, State: Initialized, Role: FOLLOWER
I20260809 15:01:58.219053 8104 consensus_queue.cc:260] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.219913 8104 ts_tablet_manager.cc:1428] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 15:01:58.223067 7437 heartbeater.cc:503] Master 127.27.186.188:40181 was elected leader, sending a full tablet report...
I20260809 15:01:58.228991 8107 tablet_bootstrap.cc:492] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Bootstrap starting.
I20260809 15:01:58.230499 8107 tablet_bootstrap.cc:654] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:58.230892 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.220369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:41479)
0809 15:01:58.230484 (+ 10115us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.230638 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9991,"thread_start_us":9828,"threads_started":1}
I20260809 15:01:58.231508 8111 tablet_bootstrap.cc:492] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Bootstrap starting.
I20260809 15:01:58.232091 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.231722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:46757)
0809 15:01:58.231871 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.231954 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 15:01:58.233291 8111 tablet_bootstrap.cc:654] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Neither blocks nor log segments found. Creating new log.
I20260809 15:01:58.233457 8107 tablet_bootstrap.cc:492] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: No bootstrap required, opened a new log
I20260809 15:01:58.233676 8107 ts_tablet_manager.cc:1397] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 15:01:58.234561 8107 raft_consensus.cc:348] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.234782 8107 raft_consensus.cc:374] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:58.234915 8107 raft_consensus.cc:729] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065727ab9d7f4b61bda3adb6a82a66ac, State: Initialized, Role: FOLLOWER
I20260809 15:01:58.235229 8107 consensus_queue.cc:260] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.235438 8111 tablet_bootstrap.cc:492] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: No bootstrap required, opened a new log
I20260809 15:01:58.235569 8111 ts_tablet_manager.cc:1397] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 15:01:58.236050 8107 ts_tablet_manager.cc:1428] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 15:01:58.236842 8111 raft_consensus.cc:348] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.237100 8111 raft_consensus.cc:374] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:01:58.237195 8111 raft_consensus.cc:729] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c021a3a60d38450381b12668e655c4df, State: Initialized, Role: FOLLOWER
I20260809 15:01:58.237454 8111 consensus_queue.cc:260] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.238225 7683 heartbeater.cc:503] Master 127.27.186.188:40181 was elected leader, sending a full tablet report...
I20260809 15:01:58.239687 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.236336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:51123)
0809 15:01:58.238520 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.238686 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1788,"thread_start_us":1648,"threads_started":1}
I20260809 15:01:58.240592 8111 ts_tablet_manager.cc:1428] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260809 15:01:58.240702 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.240355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:58435)
0809 15:01:58.240488 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.240581 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 15:01:58.242246 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.241390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:60705)
0809 15:01:58.241861 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.241984 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260809 15:01:58.242741 7931 heartbeater.cc:503] Master 127.27.186.188:40181 was elected leader, sending a full tablet report...
I20260809 15:01:58.243592 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.243245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:56553)
0809 15:01:58.243432 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.243493 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 15:01:58.373914 8120 raft_consensus.cc:482] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:01:58.374148 8120 raft_consensus.cc:504] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.375690 8120 leader_election.cc:288] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657), 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779)
I20260809 15:01:58.381577 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.375906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:36657 (local address 127.27.186.131:51029)
0809 15:01:58.376239 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.376261 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.376442 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.376822 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.376827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.376853 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.377119 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.377132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.378274 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.378280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.379705 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.379718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.380826 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.380832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.381062 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.381091 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.381172 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.381273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":28}
I20260809 15:01:58.381599 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.376025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:51029 (local address 127.27.186.129:36657)
0809 15:01:58.376263 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.376270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.376357 (+ 87us) server_negotiation.cc:407] Connection header received
0809 15:01:58.376604 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.376609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.376687 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.376788 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.377263 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.378127 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.379863 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.380639 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.380758 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.380821 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.381230 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.381361 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:01:58.381719 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.376089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:58391 (local address 127.27.186.130:35779)
0809 15:01:58.376262 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.376268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.376792 (+ 524us) server_negotiation.cc:407] Connection header received
0809 15:01:58.376952 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.376958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.377031 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.377124 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:01:58.377621 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.378368 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.379997 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.380802 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.380953 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.381022 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.381340 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.381467 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:01:58.382031 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.375906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:35779 (local address 127.27.186.131:58391)
0809 15:01:58.376689 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.376710 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:01:58.376824 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:01:58.377158 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:01:58.377161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:01:58.377181 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:01:58.377475 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.377486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.378510 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.378513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.379852 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:01:58.379867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.380975 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.380980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:01:58.381179 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.381210 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:01:58.381307 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 15:01:58.381417 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":105,"thread_start_us":425,"threads_started":1}
I20260809 15:01:58.382494 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ffd43c491745cb83b055770b35b6e5" candidate_uuid: "c021a3a60d38450381b12668e655c4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9d3ce19fa54354a136d05df2e352d0" is_pre_election: true
I20260809 15:01:58.382848 7348 raft_consensus.cc:2454] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c021a3a60d38450381b12668e655c4df in term 0.
I20260809 15:01:58.382797 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ffd43c491745cb83b055770b35b6e5" candidate_uuid: "c021a3a60d38450381b12668e655c4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" is_pre_election: true
I20260809 15:01:58.383193 7604 raft_consensus.cc:2454] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c021a3a60d38450381b12668e655c4df in term 0.
I20260809 15:01:58.383356 7731 leader_election.cc:302] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c021a3a60d38450381b12668e655c4df, cc9d3ce19fa54354a136d05df2e352d0; no voters:
I20260809 15:01:58.383729 8120 raft_consensus.cc:2781] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:01:58.383838 8120 raft_consensus.cc:482] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:01:58.383929 8120 raft_consensus.cc:3037] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:58.385985 8120 raft_consensus.cc:504] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.386556 8120 leader_election.cc:288] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [CANDIDATE]: Term 1 election: Requested vote from peers cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657), 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779)
I20260809 15:01:58.386919 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ffd43c491745cb83b055770b35b6e5" candidate_uuid: "c021a3a60d38450381b12668e655c4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9d3ce19fa54354a136d05df2e352d0"
I20260809 15:01:58.387081 7348 raft_consensus.cc:3037] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:58.387040 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ffd43c491745cb83b055770b35b6e5" candidate_uuid: "c021a3a60d38450381b12668e655c4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065727ab9d7f4b61bda3adb6a82a66ac"
I20260809 15:01:58.387212 7604 raft_consensus.cc:3037] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:01:58.388993 7348 raft_consensus.cc:2454] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c021a3a60d38450381b12668e655c4df in term 1.
I20260809 15:01:58.389217 7604 raft_consensus.cc:2454] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c021a3a60d38450381b12668e655c4df in term 1.
I20260809 15:01:58.389328 7731 leader_election.cc:302] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c021a3a60d38450381b12668e655c4df, cc9d3ce19fa54354a136d05df2e352d0; no voters:
I20260809 15:01:58.389616 8120 raft_consensus.cc:2781] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:01:58.390003 8120 raft_consensus.cc:686] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 1 LEADER]: Becoming Leader. State: Replica: c021a3a60d38450381b12668e655c4df, State: Running, Role: LEADER
I20260809 15:01:58.390416 8120 consensus_queue.cc:237] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } }
I20260809 15:01:58.390928 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.390539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:52701)
0809 15:01:58.390690 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.390790 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 15:01:58.393939 7146 catalog_manager.cc:5495] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df reported cstate change: term changed from 0 to 1, leader changed from <none> to c021a3a60d38450381b12668e655c4df (127.27.186.131). New cstate: current_term: 1 leader_uuid: "c021a3a60d38450381b12668e655c4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c021a3a60d38450381b12668e655c4df" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 } health_report { overall_health: UNKNOWN } } }
W20260809 15:01:58.398531 7442 tablet.cc:2369] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:01:58.447950 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.443472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.27.186.131:35347)
0809 15:01:58.443641 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.443645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.443664 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:01:58.443851 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.443854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.443934 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.444048 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:01:58.444330 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.445017 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.446213 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.446622 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.446710 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.447044 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:01:58.447508 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:01:58.447614 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.447658 (+ 44us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.447715 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:01:58.451545 7348 raft_consensus.cc:1261] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Refusing update from remote peer c021a3a60d38450381b12668e655c4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:58.451833 7604 raft_consensus.cc:1261] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Refusing update from remote peer c021a3a60d38450381b12668e655c4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:01:58.452219 8126 consensus_queue.cc:1035] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9d3ce19fa54354a136d05df2e352d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:58.452561 8120 consensus_queue.cc:1035] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "065727ab9d7f4b61bda3adb6a82a66ac" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:01:58.455090 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.454397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:53399)
0809 15:01:58.454767 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.454906 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260809 15:01:58.455581 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.455069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:41781)
0809 15:01:58.455306 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.455415 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260809 15:01:58.459226 8131 mvcc.cc:205] Tried to move back new op lower bound from 7316634494771564544 to 7316634494531207168. Current Snapshot: MvccSnapshot[applied={T|T < 7316634494771564544}]
I20260809 15:01:58.459584 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.459136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.130:36505)
0809 15:01:58.459360 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.459448 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260809 15:01:58.461877 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.461290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.129:41169)
0809 15:01:58.461663 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.461748 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260809 15:01:58.462379 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.461381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.27.186.131:51325)
0809 15:01:58.462114 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.462217 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618}
I20260809 15:01:58.473318 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:58.473503 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:01:58.479053 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 525c52b6bdc2459db16a822408954503: Sending DeleteTablet for 3 replicas of tablet f0ffd43c491745cb83b055770b35b6e5
I20260809 15:01:58.479964 7322 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ffd43c491745cb83b055770b35b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:58 UTC) from {username='oryx', principal='oryx/127.27.186.188@KRBTEST.COM'} at 127.0.0.1:35860
I20260809 15:01:58.479977 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ffd43c491745cb83b055770b35b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:58 UTC) from {username='oryx', principal='oryx/127.27.186.188@KRBTEST.COM'} at 127.0.0.1:50438
I20260809 15:01:58.480347 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ffd43c491745cb83b055770b35b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:01:58 UTC) from {username='oryx', principal='oryx/127.27.186.188@KRBTEST.COM'} at 127.0.0.1:35442
I20260809 15:01:58.480598 8147 tablet_replica.cc:330] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: stopping tablet replica
I20260809 15:01:58.480839 8147 raft_consensus.cc:2229] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:58.481169 8147 raft_consensus.cc:2258] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:58.482391 8147 ts_tablet_manager.cc:1905] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:58.485641 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 7106
I20260809 15:01:58.492748 8149 tablet_replica.cc:330] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: stopping tablet replica
I20260809 15:01:58.492933 8149 raft_consensus.cc:2229] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:01:58.493470 8149 raft_consensus.cc:2258] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:58.494818 8149 ts_tablet_manager.cc:1905] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:58.499770 8147 ts_tablet_manager.cc:1918] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:58.501902 8149 ts_tablet_manager.cc:1918] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:58.502077 8149 log.cc:1196] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/f0ffd43c491745cb83b055770b35b6e5
I20260809 15:01:58.502884 8147 log.cc:1196] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/f0ffd43c491745cb83b055770b35b6e5
I20260809 15:01:58.503232 8148 tablet_replica.cc:330] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: stopping tablet replica
I20260809 15:01:58.503409 8147 ts_tablet_manager.cc:1939] T f0ffd43c491745cb83b055770b35b6e5 P cc9d3ce19fa54354a136d05df2e352d0: Deleting consensus metadata
I20260809 15:01:58.503418 8148 raft_consensus.cc:2229] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:01:58.503688 8148 raft_consensus.cc:2258] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:01:58.504503 7127 catalog_manager.cc:4841] TS cc9d3ce19fa54354a136d05df2e352d0 (127.27.186.129:36657): tablet f0ffd43c491745cb83b055770b35b6e5 (table test-table [id=068d474e67e94b57b28f666766263cc5]) successfully deleted
I20260809 15:01:58.504750 8148 ts_tablet_manager.cc:1905] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:01:58.509959 8148 ts_tablet_manager.cc:1918] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:01:58.510111 8148 log.cc:1196] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/f0ffd43c491745cb83b055770b35b6e5
I20260809 15:01:58.510553 8148 ts_tablet_manager.cc:1939] T f0ffd43c491745cb83b055770b35b6e5 P 065727ab9d7f4b61bda3adb6a82a66ac: Deleting consensus metadata
I20260809 15:01:58.511514 8149 ts_tablet_manager.cc:1939] T f0ffd43c491745cb83b055770b35b6e5 P c021a3a60d38450381b12668e655c4df: Deleting consensus metadata
I20260809 15:01:58.511610 7129 catalog_manager.cc:4841] TS 065727ab9d7f4b61bda3adb6a82a66ac (127.27.186.130:35779): tablet f0ffd43c491745cb83b055770b35b6e5 (table test-table [id=068d474e67e94b57b28f666766263cc5]) successfully deleted
I20260809 15:01:58.512580 7131 catalog_manager.cc:4841] TS c021a3a60d38450381b12668e655c4df (127.27.186.131:35347): tablet f0ffd43c491745cb83b055770b35b6e5 (table test-table [id=068d474e67e94b57b28f666766263cc5]) successfully deleted
I20260809 15:01:58.518941 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.486610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.27.186.129:36657)
0809 15:01:58.486770 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.486777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.502489 (+ 15712us) server_negotiation.cc:407] Connection header received
0809 15:01:58.502660 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.502664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.502735 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.502834 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:58.505952 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.507068 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.508748 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.509366 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.509494 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.512672 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:58.512689 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:58.512707 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:58.512745 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:58.516333 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:58.516955 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:58.516965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:58.516968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:58.517123 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:58.517442 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:58.517447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:58.517450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:58.517887 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:58.518220 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.518375 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.518584 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:01:58.519544 7425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:58.560103 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.559718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37868)
0809 15:01:58.559880 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.559956 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 15:01:58.561038 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.560693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37882)
0809 15:01:58.560837 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.560929 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 15:01:58.640712 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 7106
I20260809 15:01:58.673887 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 7466
I20260809 15:01:58.694078 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.674644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.27.186.130:35779)
0809 15:01:58.674810 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.674815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.674887 (+ 72us) server_negotiation.cc:407] Connection header received
0809 15:01:58.675078 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.675082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.675148 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.675246 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:58.677132 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.677849 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.679595 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.679969 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.680057 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.688558 (+ 8501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:58.688573 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:58.688590 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:58.688621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:58.691389 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:58.692130 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:58.692137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:58.692139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:58.692281 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:58.692678 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:58.692681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:58.692683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:58.693023 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:58.693287 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.693620 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.693789 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:01:58.694727 7674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:58.808919 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 7466
I20260809 15:01:58.837247 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 7702
I20260809 15:01:58.852489 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.838009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.27.186.131:35347)
0809 15:01:58.838192 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 15:01:58.838196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:01:58.838239 (+ 43us) server_negotiation.cc:407] Connection header received
0809 15:01:58.838451 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:01:58.838454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:01:58.838533 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:01:58.838622 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:01:58.840540 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.841343 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.843097 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:01:58.843528 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:01:58.843615 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:01:58.846798 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:01:58.846810 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:01:58.846825 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:01:58.846858 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:01:58.849742 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:58.850532 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:58.850538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:58.850540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:58.850692 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:01:58.851118 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:01:58.851121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:01:58.851123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:01:58.851476 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:01:58.851693 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 15:01:58.852042 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 15:01:58.852210 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:01:58.853201 7925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:58.945639 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.945171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37886)
0809 15:01:58.945355 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.945464 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 15:01:58.946688 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:01:58.946305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36429 (local address 127.0.0.1:37890)
0809 15:01:58.946445 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 15:01:58.946532 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 15:01:58.978610 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 7702
I20260809 15:01:59.007973 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 6602
I20260809 15:01:59.009127 6881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:59.121270 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 6602
I20260809 15:01:59.151283 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 6768
I20260809 15:01:59.152351 7193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:01:59.272460 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 6768
2026-08-09T15:01:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9400 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:01:59 dist-test-slave-q9h9 krb5kdc[8166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:01:59 dist-test-slave-q9h9 krb5kdc[8166](info): set up 2 sockets
Aug 09 15:01:59 dist-test-slave-q9h9 krb5kdc[8166](info): commencing operation
krb5kdc: starting...
W20260809 15:02:00.480913 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:02:00 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab.
Entry for principal oryx/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab.
I20260809 15:02:00.673302 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43771
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab
--principal=oryx/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:43771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:00.971935 8188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:00.972182 8188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:00.972289 8188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:00.979058 8188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:00.979139 8188 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 15:02:00.979192 8188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:00.979271 8188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:00.979329 8188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:00.979411 8188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:00.988435 8188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:43771
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.190.keytab
--principal=oryx/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:00.989595 8188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:00.990796 8188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:01.001314 8197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:01.001384 8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:01.001354 8205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:01.002380 8188 server_base.cc:1034] running on GCE node
I20260809 15:02:01.003002 8188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:01.004192 8188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:01.005393 8188 hybrid_clock.cc:648] HybridClock initialized: now 1786287721005336 us; error 36 us; skew 500 ppm
I20260809 15:02:01.005681 8188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:01 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287721, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:01.008574 8188 init.cc:377] Logged in from keytab as oryx/127.27.186.190@KRBTEST.COM (short username oryx)
I20260809 15:02:01.009903 8188 webserver.cc:460] Webserver started at http://127.27.186.190:40115/ using document root <none> and password file <none>
I20260809 15:02:01.010535 8188 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:01.010620 8188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:01.010888 8188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:01.013547 8188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "02f0ba040f354a2d962cb1e98c859d2c"
format_stamp: "Formatted at 2026-08-09 15:02:01 on dist-test-slave-q9h9"
server_key: "7a39d53036d407c486e6f4bbabbc1918"
server_key_iv: "fbc9b81fe954cafa191186812da90928"
server_key_version: "encryptionkey@0"
I20260809 15:02:01.014304 8188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "02f0ba040f354a2d962cb1e98c859d2c"
format_stamp: "Formatted at 2026-08-09 15:02:01 on dist-test-slave-q9h9"
server_key: "7a39d53036d407c486e6f4bbabbc1918"
server_key_iv: "fbc9b81fe954cafa191186812da90928"
server_key_version: "encryptionkey@0"
I20260809 15:02:01.018980 8188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:02:01.022143 8219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:01.023335 8188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:02:01.023497 8188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "02f0ba040f354a2d962cb1e98c859d2c"
format_stamp: "Formatted at 2026-08-09 15:02:01 on dist-test-slave-q9h9"
server_key: "7a39d53036d407c486e6f4bbabbc1918"
server_key_iv: "fbc9b81fe954cafa191186812da90928"
server_key_version: "encryptionkey@0"
I20260809 15:02:01.023685 8188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:01.049119 8188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:01.052664 8188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:01.052887 8188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:01.072599 8188 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:43771
I20260809 15:02:01.072643 8324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:43771 every 8 connection(s)
I20260809 15:02:01.073741 8188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:02:01.077327 8325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:01.079862 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 8188
I20260809 15:02:01.080082 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:02:01.080978 28394 external_mini_cluster.cc:1428] Setting key 5013ff1a1cfe2deeacccde9181963332
I20260809 15:02:01.089100 8325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Bootstrap starting.
I20260809 15:02:01.092229 8325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:01.093248 8325 log.cc:824] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:02:01 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:01.095760 8325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: No bootstrap required, opened a new log
I20260809 15:02:01.104597 8325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02f0ba040f354a2d962cb1e98c859d2c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43771 } }
I20260809 15:02:01.105047 8325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:01.105232 8325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f0ba040f354a2d962cb1e98c859d2c, State: Initialized, Role: FOLLOWER
I20260809 15:02:01.105803 8325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02f0ba040f354a2d962cb1e98c859d2c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43771 } }
I20260809 15:02:01.106103 8325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:02:01.106256 8325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:02:01.106313 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:01.083151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.27.186.190:43771)
0809 15:02:01.083923 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:01.083933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:01.083970 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:02:01.085613 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:01.085634 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:01.085984 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:01.086373 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:01.087862 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.089169 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.090448 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.091091 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.091267 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:01.099288 (+ 8021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:01.099296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:01.099312 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:01.099347 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:01.102247 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:01.103141 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:01.103146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:01.103152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:01.103325 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:01.103724 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:01.103727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:01.103729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:01.104184 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:01.104418 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:01.104864 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 15:02:01.105202 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":355,"threads_started":1}
I20260809 15:02:01.106448 8325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:01.109210 8325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02f0ba040f354a2d962cb1e98c859d2c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43771 } }
I20260809 15:02:01.109690 8325 leader_election.cc:302] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02f0ba040f354a2d962cb1e98c859d2c; no voters:
I20260809 15:02:01.110275 8325 leader_election.cc:288] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:02:01.110352 8331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:01.111040 8331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [term 1 LEADER]: Becoming Leader. State: Replica: 02f0ba040f354a2d962cb1e98c859d2c, State: Running, Role: LEADER
I20260809 15:02:01.111575 8331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02f0ba040f354a2d962cb1e98c859d2c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43771 } }
I20260809 15:02:01.112219 8325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:02:01.114446 8333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f0ba040f354a2d962cb1e98c859d2c. Latest consensus state: current_term: 1 leader_uuid: "02f0ba040f354a2d962cb1e98c859d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02f0ba040f354a2d962cb1e98c859d2c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43771 } } }
I20260809 15:02:01.114840 8333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:01.114882 8332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02f0ba040f354a2d962cb1e98c859d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02f0ba040f354a2d962cb1e98c859d2c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43771 } } }
I20260809 15:02:01.115269 8332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:01.117101 8340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:02:01.122215 8340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:02:01.131206 8340 catalog_manager.cc:1347] Generated new cluster ID: fd464b7c4cdd4fdc82129f6d8cd62ced
I20260809 15:02:01.131300 8340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:02:01.138523 8340 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:02:01.139658 8340 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:02:01.150312 8340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Generated new TSK 0
I20260809 15:02:01.151180 8340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab.
Entry for principal oryx/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab.
I20260809 15:02:01.209041 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43771
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:01.524374 8364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:01.524626 8364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:01.524765 8364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:01.531872 8364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:01.531978 8364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:01.532095 8364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:02:01.532208 8364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:02:01.541263 8364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:43771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:01.542351 8364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:01.543524 8364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:01.555151 8375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:01.555234 8377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:01.555192 8380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:01.556221 8364 server_base.cc:1034] running on GCE node
I20260809 15:02:01.556773 8364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:01.557577 8364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:01.558777 8364 hybrid_clock.cc:648] HybridClock initialized: now 1786287721558714 us; error 37 us; skew 500 ppm
I20260809 15:02:01.559031 8364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:01 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287721, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:01.561949 8364 init.cc:377] Logged in from keytab as oryx/127.27.186.129@KRBTEST.COM (short username oryx)
I20260809 15:02:01.563218 8364 webserver.cc:460] Webserver started at http://127.27.186.129:38593/ using document root <none> and password file <none>
I20260809 15:02:01.563808 8364 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:01.563905 8364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:01.564266 8364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:01.566915 8364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "6c4fada8d4024784aead7c57fa300b27"
format_stamp: "Formatted at 2026-08-09 15:02:01 on dist-test-slave-q9h9"
server_key: "5c7645802ce8a74b101d797973019068"
server_key_iv: "f0b9b80513a03038f0a50dd7ec403bf5"
server_key_version: "encryptionkey@0"
I20260809 15:02:01.567648 8364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "6c4fada8d4024784aead7c57fa300b27"
format_stamp: "Formatted at 2026-08-09 15:02:01 on dist-test-slave-q9h9"
server_key: "5c7645802ce8a74b101d797973019068"
server_key_iv: "f0b9b80513a03038f0a50dd7ec403bf5"
server_key_version: "encryptionkey@0"
I20260809 15:02:01.572191 8364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:02:01.575296 8391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:01.576424 8364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 15:02:01.576565 8364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "6c4fada8d4024784aead7c57fa300b27"
format_stamp: "Formatted at 2026-08-09 15:02:01 on dist-test-slave-q9h9"
server_key: "5c7645802ce8a74b101d797973019068"
server_key_iv: "f0b9b80513a03038f0a50dd7ec403bf5"
server_key_version: "encryptionkey@0"
I20260809 15:02:01.576753 8364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:01.592409 8364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:01.596212 8364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:01.596443 8364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:01.598914 8364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:01.598996 8364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:01.599128 8364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:01.599202 8364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:02:01 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287721, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:01.619241 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:01.599441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:58115 (local address 127.27.186.190:43771)
0809 15:02:01.599725 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:01.599731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:01.600932 (+ 1201us) server_negotiation.cc:407] Connection header received
0809 15:02:01.602967 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:01.602973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:01.603064 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:01.603176 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:01.605214 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.606215 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.608626 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.609084 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.609201 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:01.613303 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:01.613310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:01.613315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:01.613349 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:01.616139 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:01.616928 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:01.616933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:01.616937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:01.617069 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:01.617520 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:01.617525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:01.617529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:01.617845 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:01.617987 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:01.618682 (+ 695us) server_negotiation.cc:299] Negotiation successful
0809 15:02:01.618919 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260809 15:02:01.620128 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:01.599912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.27.186.129:58115)
0809 15:02:01.600743 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:01.600797 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:01.602680 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:01.603442 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:01.603453 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:01.604159 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:01.604953 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:01.604984 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.606363 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.606367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:01.608470 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:01.608490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.609355 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.609364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:01.609536 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:01.610164 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:01.610185 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:01.613052 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:01.616270 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:01.616278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:01.616293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:01.616782 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:01.617199 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:01.617204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:01.617208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:01.617410 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:01.617968 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:01.617976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:01.618352 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 15:02:01.618574 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 15:02:01.619057 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":355,"threads_started":1}
I20260809 15:02:01.642318 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:01.629985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:37933 (local address 127.27.186.190:43771)
0809 15:02:01.630172 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:01.630178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:01.630195 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:02:01.630240 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:01.630244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:01.630309 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:01.630393 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:01.631640 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.632689 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.634661 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.635307 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.635403 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:01.637670 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:01.637677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:01.637682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:01.637713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:01.639911 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:01.640504 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:01.640509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:01.640514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:01.640633 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:01.640967 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:01.640972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:01.640976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:01.641232 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:01.641370 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:01.641749 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 15:02:01.641947 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:02:01.642454 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:01.628825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.27.186.129:37933)
0809 15:02:01.629634 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:01.629652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:01.629759 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:01.630552 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:01.630557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:01.631072 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:01.631462 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:01.631476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.632905 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.632910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:01.634494 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:01.634510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:01.635702 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:01.635709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:01.635960 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:01.636541 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:01.636564 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:01.637506 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:01.640032 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:01.640037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:01.640041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:01.640396 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:01.640735 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:01.640740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:01.640744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:01.640868 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:01.641351 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:01.641356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:01.641524 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 15:02:01.641638 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 15:02:01.641947 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":411,"threads_started":1}
I20260809 15:02:01.654147 8364 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:35707
I20260809 15:02:01.654155 8591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:35707 every 8 connection(s)
I20260809 15:02:01.655239 8364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:02:01.657011 8593 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43771
I20260809 15:02:01.657218 8593 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:01.657804 8593 heartbeater.cc:511] Master 127.27.186.190:43771 requested a full tablet report, sending...
I20260809 15:02:01.659520 8245 ts_manager.cc:194] Registered new tserver with Master: 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707)
I20260809 15:02:01.660880 8245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:58115
I20260809 15:02:01.661530 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 8364
I20260809 15:02:01.661731 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:02:01.662400 28394 external_mini_cluster.cc:1428] Setting key 765c6faa06c28d613a375353592bba42
WARNING: no policy specified for oryx/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab.
Entry for principal oryx/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab.
I20260809 15:02:01.706418 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43771
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:02.015780 8598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:02.016009 8598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:02.016131 8598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:02.023102 8598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:02.023204 8598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:02.023324 8598 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:02:02.023422 8598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:02:02.032717 8598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:43771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:02.033790 8598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:02.034987 8598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:02.046751 8609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:02.046820 8607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:02.046983 8605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:02.047755 8598 server_base.cc:1034] running on GCE node
I20260809 15:02:02.048282 8598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:02.049059 8598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:02.050264 8598 hybrid_clock.cc:648] HybridClock initialized: now 1786287722050206 us; error 31 us; skew 500 ppm
I20260809 15:02:02.050560 8598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287722, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:02.053494 8598 init.cc:377] Logged in from keytab as oryx/127.27.186.130@KRBTEST.COM (short username oryx)
I20260809 15:02:02.054766 8598 webserver.cc:460] Webserver started at http://127.27.186.130:35541/ using document root <none> and password file <none>
I20260809 15:02:02.055364 8598 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:02.055447 8598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:02.055706 8598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:02.058434 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "c710fa40b9b1496f92c4454a10d47e0f"
format_stamp: "Formatted at 2026-08-09 15:02:02 on dist-test-slave-q9h9"
server_key: "b471663b6841e5bb5f74adfc76e12680"
server_key_iv: "c7b94d0ab3fdb8430aee74afbcf51f8a"
server_key_version: "encryptionkey@0"
I20260809 15:02:02.059128 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "c710fa40b9b1496f92c4454a10d47e0f"
format_stamp: "Formatted at 2026-08-09 15:02:02 on dist-test-slave-q9h9"
server_key: "b471663b6841e5bb5f74adfc76e12680"
server_key_iv: "c7b94d0ab3fdb8430aee74afbcf51f8a"
server_key_version: "encryptionkey@0"
I20260809 15:02:02.063865 8598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:02:02.066982 8625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:02.068120 8598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:02:02.068265 8598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "c710fa40b9b1496f92c4454a10d47e0f"
format_stamp: "Formatted at 2026-08-09 15:02:02 on dist-test-slave-q9h9"
server_key: "b471663b6841e5bb5f74adfc76e12680"
server_key_iv: "c7b94d0ab3fdb8430aee74afbcf51f8a"
server_key_version: "encryptionkey@0"
I20260809 15:02:02.068463 8598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:02.100558 8598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:02.103979 8598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:02.104223 8598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:02.106608 8598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:02.106709 8598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:02.106832 8598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:02.106909 8598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287722, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:02.129647 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.107198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:34499 (local address 127.27.186.190:43771)
0809 15:02:02.107368 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.107372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.108897 (+ 1525us) server_negotiation.cc:407] Connection header received
0809 15:02:02.110853 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.110859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.110940 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.111042 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:02.113249 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.114291 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.117023 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.117556 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.117709 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.122576 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:02.122582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:02.122587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:02.122627 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:02.125660 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.126734 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.126739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.126743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.126879 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.127646 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.127652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.127655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.128029 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:02.128166 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.129091 (+ 925us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.129330 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:02:02.130892 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.107997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.27.186.130:34499)
0809 15:02:02.108741 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.108786 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.110590 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.111315 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.111330 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.112146 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:02.113031 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.113062 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.114452 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.114458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.116835 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.116870 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.117835 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.117843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.118207 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.119050 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:02.119085 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:02.122321 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:02.125849 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.125869 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.125888 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.126591 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.127300 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.127304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.127306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.127524 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.128157 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:02.128164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:02.128614 (+ 450us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.128972 (+ 358us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.129481 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
I20260809 15:02:02.148581 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.135705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:47189 (local address 127.27.186.190:43771)
0809 15:02:02.135885 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.135891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.136488 (+ 597us) server_negotiation.cc:407] Connection header received
0809 15:02:02.136640 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.136644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.136696 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.136767 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:02.137931 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.138875 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.140117 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.140639 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.140764 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.142869 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:02.142873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:02.142876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:02.142903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:02.145900 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.146539 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.146544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.146548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.146693 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.147037 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.147041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.147044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.147435 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:02.147587 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.148031 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.148231 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:02:02.148674 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.135766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.27.186.130:47189)
0809 15:02:02.136382 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.136401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.136536 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.136804 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.136809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.137347 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:02.137762 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.137778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.139000 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.139005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.139987 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.139999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.140839 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.140847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.141036 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.141612 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:02.141640 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:02.142676 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:02.146049 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.146054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.146058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.146441 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.146782 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.146785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.146787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.146942 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.147568 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:02.147574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:02.147792 (+ 218us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.147931 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.148271 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":311,"threads_started":1}
I20260809 15:02:02.159932 8598 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:39871
I20260809 15:02:02.159961 8850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:39871 every 8 connection(s)
I20260809 15:02:02.160987 8598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:02:02.162276 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 8598
I20260809 15:02:02.162449 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:02:02.162770 8853 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43771
I20260809 15:02:02.162947 28394 external_mini_cluster.cc:1428] Setting key 9e5b4c11426bcf91755e87d65ccb0caa
I20260809 15:02:02.162974 8853 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:02.163538 8853 heartbeater.cc:511] Master 127.27.186.190:43771 requested a full tablet report, sending...
I20260809 15:02:02.164860 8245 ts_manager.cc:194] Registered new tserver with Master: c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871)
I20260809 15:02:02.165623 8245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:34499
WARNING: no policy specified for oryx/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab.
Entry for principal oryx/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab.
I20260809 15:02:02.209259 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43771
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:02.516776 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:02.517030 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:02.517164 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:02.524127 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:02.524256 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:02.524371 8862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 15:02:02.524479 8862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:02:02.533799 8862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.186.190:43771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:02.534875 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:02.536042 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:02.547871 8874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:02.547899 8875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:02.547825 8877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:02.548583 8862 server_base.cc:1034] running on GCE node
I20260809 15:02:02.549147 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:02.549932 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:02.551136 8862 hybrid_clock.cc:648] HybridClock initialized: now 1786287722551058 us; error 52 us; skew 500 ppm
I20260809 15:02:02.551447 8862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287722, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:02.554378 8862 init.cc:377] Logged in from keytab as oryx/127.27.186.131@KRBTEST.COM (short username oryx)
I20260809 15:02:02.555619 8862 webserver.cc:460] Webserver started at http://127.27.186.131:41425/ using document root <none> and password file <none>
I20260809 15:02:02.556221 8862 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:02.556316 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:02.556609 8862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:02.559254 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "c8d591e1805f4250a81903db6c16a9c5"
format_stamp: "Formatted at 2026-08-09 15:02:02 on dist-test-slave-q9h9"
server_key: "6c2c321f74e9b11bab70780595add6ac"
server_key_iv: "166d858a890d760d7057d1a782a80dfd"
server_key_version: "encryptionkey@0"
I20260809 15:02:02.560128 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "c8d591e1805f4250a81903db6c16a9c5"
format_stamp: "Formatted at 2026-08-09 15:02:02 on dist-test-slave-q9h9"
server_key: "6c2c321f74e9b11bab70780595add6ac"
server_key_iv: "166d858a890d760d7057d1a782a80dfd"
server_key_version: "encryptionkey@0"
I20260809 15:02:02.564851 8862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:02:02.568028 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:02.569186 8862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:02:02.569327 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "c8d591e1805f4250a81903db6c16a9c5"
format_stamp: "Formatted at 2026-08-09 15:02:02 on dist-test-slave-q9h9"
server_key: "6c2c321f74e9b11bab70780595add6ac"
server_key_iv: "166d858a890d760d7057d1a782a80dfd"
server_key_version: "encryptionkey@0"
I20260809 15:02:02.569507 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:02.581753 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:02.585148 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:02.585376 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:02.587726 8862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:02.587809 8862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:02.587929 8862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:02.587992 8862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287722, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:02.609753 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.588245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:53321 (local address 127.27.186.190:43771)
0809 15:02:02.588433 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.588439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.589688 (+ 1249us) server_negotiation.cc:407] Connection header received
0809 15:02:02.591623 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.591629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.591741 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.591864 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:02.593850 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.594995 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.597258 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.597707 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.597834 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.602609 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:02.602616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:02.602620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:02.602669 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:02.605678 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.606616 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.606621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.606625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.606776 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.607357 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.607362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.607366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.607723 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:02.607890 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.609021 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.609320 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 15:02:02.610472 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.588716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.27.186.131:53321)
0809 15:02:02.589493 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.589538 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.591352 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.592106 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.592117 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.592813 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:02.593599 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.593623 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.595145 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.595149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.597111 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.597123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.597978 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.597987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.598345 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.599067 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:02.599095 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:02.602319 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:02.605845 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.605852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.605862 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.606501 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.606892 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.606896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.606898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.607203 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.607972 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:02.607977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:02.608505 (+ 528us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.608853 (+ 348us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.609293 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":341,"threads_started":1}
I20260809 15:02:02.630671 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.616249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.27.186.131:60609)
0809 15:02:02.616820 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.616835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.616975 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.617312 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.617316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.617747 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:02.618103 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.618113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.619325 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.619330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.620971 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.620986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.621906 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.621912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.622104 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.622894 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:02.622921 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:02.623890 (+ 969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:02.627531 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.627540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.627543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.628159 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.628640 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:02.628645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:02.628648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:02.628850 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:02.629462 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:02.629468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:02.629686 (+ 218us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.629858 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.630215 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":304,"threads_started":1}
I20260809 15:02:02.630858 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.616152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:60609 (local address 127.27.186.190:43771)
0809 15:02:02.616296 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.616300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.616928 (+ 628us) server_negotiation.cc:407] Connection header received
0809 15:02:02.617092 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.617095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.617185 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.617263 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:02.618243 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.619198 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.621127 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.621696 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.621869 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.624071 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:02.624081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:02.624085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:02.624113 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:02.627413 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.628275 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.628280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.628284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.628473 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.628977 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.628982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.628985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.629254 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:02.629394 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.630296 (+ 902us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.630504 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 15:02:02.633605 8862 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:33821
I20260809 15:02:02.633635 9075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:33821 every 8 connection(s)
I20260809 15:02:02.634809 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:02:02.635421 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 8862
I20260809 15:02:02.635603 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:02:02.636168 28394 external_mini_cluster.cc:1428] Setting key 460618355ec39b31815a522fbf87fc86
I20260809 15:02:02.636830 9078 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43771
I20260809 15:02:02.637176 9078 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:02.637828 9078 heartbeater.cc:511] Master 127.27.186.190:43771 requested a full tablet report, sending...
I20260809 15:02:02.639052 8245 ts_manager.cc:194] Registered new tserver with Master: c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821)
I20260809 15:02:02.639791 8245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:53321
I20260809 15:02:02.650806 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:02:02.663302 8593 heartbeater.cc:503] Master 127.27.186.190:43771 was elected leader, sending a full tablet report...
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
W20260809 15:02:02.700736 8329 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:02.691335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.27.186.190:43771)
0809 15:02:02.691491 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.691495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.691807 (+ 312us) server_negotiation.cc:407] Connection header received
0809 15:02:02.691946 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.691949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.692014 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.692093 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:02.693311 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.693989 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.695153 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.695550 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.695635 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.700468 (+ 4833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36846: BlockingRecv error: recv got EOF from 127.0.0.1:36846 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 15:02:02 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:02.715430 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.703551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.27.186.190:43771)
0809 15:02:02.703692 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.703696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.703998 (+ 302us) server_negotiation.cc:407] Connection header received
0809 15:02:02.704109 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.704112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.704159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.704238 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:02.705551 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.706274 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.707332 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.707720 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.707805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.710884 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:02.710889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:02.710892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:02.710920 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:02.713321 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.713938 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.713942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.713944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.714090 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:02.714384 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:02.714387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:02.714389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:02.714651 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:02.714821 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.715039 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.715188 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:02:02.718801 8245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:02:02.725337 8245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:02:02.762338 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.736317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39871 (local address 127.0.0.1:41024)
0809 15:02:02.744842 (+ 8525us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.744881 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.745064 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.754871 (+ 9807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.754879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.754913 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:02.755283 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.755296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.757060 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.757065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.759385 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.759400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.761342 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.761348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.761525 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.761574 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.761703 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.761886 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8264,"mutex_wait_us":3,"thread_start_us":16435,"threads_started":1}
I20260809 15:02:02.762466 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.736604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:35707 (local address 127.0.0.1:45576)
0809 15:02:02.753072 (+ 16468us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.753092 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.753210 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.756198 (+ 2988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.756203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.756227 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:02.756621 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.756635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.758345 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.758350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.760251 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.760262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.761704 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.761710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.761860 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.761889 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.761975 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.762117 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16319,"mutex_wait_us":61,"thread_start_us":18818,"threads_started":1}
I20260809 15:02:02.762504 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.737041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.27.186.129:35707)
0809 15:02:02.755380 (+ 18339us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.755389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.755416 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:02:02.755513 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.755520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.755803 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.755988 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:02.756812 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.758217 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.760400 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.761497 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.761664 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.761826 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.762066 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.762226 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18178,"thread_start_us":18031,"threads_started":1}
I20260809 15:02:02.763077 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.736551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.27.186.130:39871)
0809 15:02:02.754203 (+ 17652us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.754211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.754242 (+ 31us) server_negotiation.cc:407] Connection header received
0809 15:02:02.754345 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.754353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.754599 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.754805 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:02.755494 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.756910 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.759575 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.761160 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.761319 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.761492 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.762765 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.762883 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17464,"thread_start_us":17328,"threads_started":1}
I20260809 15:02:02.764746 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.738534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.27.186.131:33821)
0809 15:02:02.744283 (+ 5749us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.744293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.755767 (+ 11474us) server_negotiation.cc:407] Connection header received
0809 15:02:02.756041 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.756050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.756246 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.756413 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:02.757294 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.758431 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.760955 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.762268 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.762829 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.762942 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.764262 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.764434 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5562,"thread_start_us":5363,"threads_started":1}
I20260809 15:02:02.765406 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.738906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33821 (local address 127.0.0.1:34100)
0809 15:02:02.755663 (+ 16757us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.755679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.755867 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.756721 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.756726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.756748 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:02.757096 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.757106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.758586 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.758592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.760794 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.760808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.763888 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.763894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.764056 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.764088 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.764934 (+ 846us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.765059 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16652,"mutex_wait_us":21,"thread_start_us":5541,"threads_started":1}
I20260809 15:02:02.766167 8472 tablet_service.cc:1467] Processing CreateTablet for tablet f5af290009ca46bd8a15a12d244248a1 (DEFAULT_TABLE table=test-table [id=04a28bf5b95545579534d1585189438d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:02.766574 8721 tablet_service.cc:1467] Processing CreateTablet for tablet f5af290009ca46bd8a15a12d244248a1 (DEFAULT_TABLE table=test-table [id=04a28bf5b95545579534d1585189438d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:02.767566 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5af290009ca46bd8a15a12d244248a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:02.767611 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5af290009ca46bd8a15a12d244248a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:02.768847 8961 tablet_service.cc:1467] Processing CreateTablet for tablet f5af290009ca46bd8a15a12d244248a1 (DEFAULT_TABLE table=test-table [id=04a28bf5b95545579534d1585189438d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:02.770161 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5af290009ca46bd8a15a12d244248a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:02.778472 9111 tablet_bootstrap.cc:492] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Bootstrap starting.
I20260809 15:02:02.781888 9111 tablet_bootstrap.cc:654] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:02.782198 9114 tablet_bootstrap.cc:492] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Bootstrap starting.
I20260809 15:02:02.783201 9111 log.cc:824] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:02.783546 9117 tablet_bootstrap.cc:492] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Bootstrap starting.
I20260809 15:02:02.785339 9114 tablet_bootstrap.cc:654] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:02.786309 9117 tablet_bootstrap.cc:654] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:02.786561 9114 log.cc:824] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:02.786659 9111 tablet_bootstrap.cc:492] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: No bootstrap required, opened a new log
I20260809 15:02:02.786917 9111 ts_tablet_manager.cc:1397] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 15:02:02.787290 9117 log.cc:824] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:02.789289 9114 tablet_bootstrap.cc:492] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: No bootstrap required, opened a new log
I20260809 15:02:02.789539 9114 ts_tablet_manager.cc:1397] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 15:02:02.790118 9117 tablet_bootstrap.cc:492] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: No bootstrap required, opened a new log
I20260809 15:02:02.790369 9117 ts_tablet_manager.cc:1397] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:02:02.794109 9111 raft_consensus.cc:348] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.794433 9111 raft_consensus.cc:374] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:02.794535 9111 raft_consensus.cc:729] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c710fa40b9b1496f92c4454a10d47e0f, State: Initialized, Role: FOLLOWER
I20260809 15:02:02.795060 9111 consensus_queue.cc:260] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.796020 8853 heartbeater.cc:503] Master 127.27.186.190:43771 was elected leader, sending a full tablet report...
I20260809 15:02:02.796610 9111 ts_tablet_manager.cc:1428] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260809 15:02:02.799562 9114 raft_consensus.cc:348] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.799963 9114 raft_consensus.cc:374] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:02.800076 9114 raft_consensus.cc:729] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4fada8d4024784aead7c57fa300b27, State: Initialized, Role: FOLLOWER
I20260809 15:02:02.800172 9117 raft_consensus.cc:348] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.800653 9117 raft_consensus.cc:374] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:02.800755 9117 raft_consensus.cc:729] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Initialized, Role: FOLLOWER
I20260809 15:02:02.801297 9117 consensus_queue.cc:260] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.801690 9114 consensus_queue.cc:260] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.802433 9078 heartbeater.cc:503] Master 127.27.186.190:43771 was elected leader, sending a full tablet report...
I20260809 15:02:02.803014 9117 ts_tablet_manager.cc:1428] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 15:02:02.803280 9114 ts_tablet_manager.cc:1428] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260809 15:02:02.870226 9127 raft_consensus.cc:482] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:02:02.870476 9127 raft_consensus.cc:504] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.872234 9127 leader_election.cc:288] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871), 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707)
I20260809 15:02:02.878159 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.872466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:40891 (local address 127.27.186.130:39871)
0809 15:02:02.872658 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.872664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.872983 (+ 319us) server_negotiation.cc:407] Connection header received
0809 15:02:02.873057 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.873062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.873131 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.873246 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:02.873643 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.874609 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.876360 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.877331 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.877446 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.877495 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.877817 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.877898 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:02:02.878218 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.872461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39871 (local address 127.27.186.131:40891)
0809 15:02:02.872794 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.872818 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.872947 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.873291 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.873294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.873317 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:02.873510 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.873522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.874748 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.874753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.876187 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.876199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.877477 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.877483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.877670 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.877705 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.877805 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.877931 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260809 15:02:02.878691 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.872755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:33569 (local address 127.27.186.129:35707)
0809 15:02:02.872926 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.872932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.873394 (+ 462us) server_negotiation.cc:407] Connection header received
0809 15:02:02.873531 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.873536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.873619 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.873689 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:02.874295 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.875148 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.876941 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.877860 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.877990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.878084 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.878422 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.878541 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:02:02.878870 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.872549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:35707 (local address 127.27.186.131:33569)
0809 15:02:02.873292 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:02.873310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:02.873429 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:02.873726 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:02.873730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:02.873754 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:02.874125 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.874137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.875283 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.875287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.876793 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:02.876807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.878088 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.878095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:02.878269 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.878302 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:02:02.878399 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 15:02:02.878517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":3,"thread_start_us":389,"threads_started":1}
I20260809 15:02:02.879197 8759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af290009ca46bd8a15a12d244248a1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f" is_pre_election: true
I20260809 15:02:02.879556 8759 raft_consensus.cc:2454] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 0.
I20260809 15:02:02.879667 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af290009ca46bd8a15a12d244248a1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27" is_pre_election: true
I20260809 15:02:02.880187 8512 raft_consensus.cc:2454] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 0.
I20260809 15:02:02.880172 8889 leader_election.cc:302] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:02.880570 9127 raft_consensus.cc:2781] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:02:02.880725 9127 raft_consensus.cc:482] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:02:02.880829 9127 raft_consensus.cc:3037] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:02.883023 9127 raft_consensus.cc:504] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.883555 9127 leader_election.cc:288] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Requested vote from peers c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871), 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707)
I20260809 15:02:02.883881 8759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af290009ca46bd8a15a12d244248a1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f"
I20260809 15:02:02.883940 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af290009ca46bd8a15a12d244248a1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27"
I20260809 15:02:02.884043 8759 raft_consensus.cc:3037] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:02.884100 8512 raft_consensus.cc:3037] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:02.886178 8759 raft_consensus.cc:2454] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 1.
I20260809 15:02:02.886180 8512 raft_consensus.cc:2454] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 1.
W20260809 15:02:02.886339 9079 tablet.cc:2369] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:02:02.886515 8889 leader_election.cc:302] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:02.886780 9127 raft_consensus.cc:2781] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:02.887233 9127 raft_consensus.cc:686] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 LEADER]: Becoming Leader. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Running, Role: LEADER
I20260809 15:02:02.887632 9127 consensus_queue.cc:237] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.891301 8243 catalog_manager.cc:5495] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } health_report { overall_health: HEALTHY } } }
W20260809 15:02:02.907285 8594 tablet.cc:2369] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:02:02.911682 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.906677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.27.186.131:33821)
0809 15:02:02.906857 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:02.906862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:02.907005 (+ 143us) server_negotiation.cc:407] Connection header received
0809 15:02:02.907215 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:02.907219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:02.907297 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:02.907421 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:02.907767 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.908445 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.909739 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:02.910179 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:02.910284 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:02.910719 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:02.911166 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:02.911286 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:02.911394 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 15:02:02.911467 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 15:02:02.912632 8854 tablet.cc:2369] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:02:02.918843 8759 raft_consensus.cc:1261] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Refusing update from remote peer c8d591e1805f4250a81903db6c16a9c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:02.918840 8512 raft_consensus.cc:1261] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Refusing update from remote peer c8d591e1805f4250a81903db6c16a9c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:02.919818 9127 consensus_queue.cc:1035] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:02.920089 9131 consensus_queue.cc:1035] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:02.929399 9135 mvcc.cc:205] Tried to move back new op lower bound from 7316634513066561536 to 7316634512950779904. Current Snapshot: MvccSnapshot[applied={T|T < 7316634513066561536}]
I20260809 15:02:02.934336 9137 mvcc.cc:205] Tried to move back new op lower bound from 7316634513066561536 to 7316634512950779904. Current Snapshot: MvccSnapshot[applied={T|T < 7316634513066561536}]
I20260809 15:02:02.934947 9139 mvcc.cc:205] Tried to move back new op lower bound from 7316634513066561536 to 7316634512950779904. Current Snapshot: MvccSnapshot[applied={T|T < 7316634513066561536}]
I20260809 15:02:02.942363 8243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:02.942590 8243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:02.945773 8243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Sending DeleteTablet for 3 replicas of tablet f5af290009ca46bd8a15a12d244248a1
I20260809 15:02:02.946767 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet f5af290009ca46bd8a15a12d244248a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:02 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:41024
I20260809 15:02:02.946846 8472 tablet_service.cc:1514] Processing DeleteTablet for tablet f5af290009ca46bd8a15a12d244248a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:02 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45576
I20260809 15:02:02.947588 9149 tablet_replica.cc:330] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: stopping tablet replica
I20260809 15:02:02.947664 8961 tablet_service.cc:1514] Processing DeleteTablet for tablet f5af290009ca46bd8a15a12d244248a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:02 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34100
I20260809 15:02:02.948076 9149 raft_consensus.cc:2229] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:02.948545 9149 raft_consensus.cc:2258] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:02.949283 9148 tablet_replica.cc:330] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: stopping tablet replica
I20260809 15:02:02.949497 8243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 15:02:02.949679 9148 raft_consensus.cc:2229] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 15:02:02.950145 8243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:02:02.950368 9148 raft_consensus.cc:2258] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:02.950896 9149 ts_tablet_manager.cc:1905] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:02.953145 9148 ts_tablet_manager.cc:1905] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:02.959843 9148 ts_tablet_manager.cc:1918] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:02.960049 9148 log.cc:1196] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/f5af290009ca46bd8a15a12d244248a1
I20260809 15:02:02.960624 9148 ts_tablet_manager.cc:1939] T f5af290009ca46bd8a15a12d244248a1 P c710fa40b9b1496f92c4454a10d47e0f: Deleting consensus metadata
I20260809 15:02:02.960757 8472 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef5f615967c4a128f510dc53ae95957 (DEFAULT_TABLE table=test-table [id=09a9fd4c1df8478b98a25260ea4afdd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:02.961392 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef5f615967c4a128f510dc53ae95957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:02.962565 8222 catalog_manager.cc:4841] TS c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871): tablet f5af290009ca46bd8a15a12d244248a1 (table test-table [id=04a28bf5b95545579534d1585189438d]) successfully deleted
I20260809 15:02:02.962910 8721 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef5f615967c4a128f510dc53ae95957 (DEFAULT_TABLE table=test-table [id=09a9fd4c1df8478b98a25260ea4afdd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:02.963299 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef5f615967c4a128f510dc53ae95957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:02.964044 8959 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef5f615967c4a128f510dc53ae95957 (DEFAULT_TABLE table=test-table [id=09a9fd4c1df8478b98a25260ea4afdd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:02.964582 8959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef5f615967c4a128f510dc53ae95957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:02.971175 9117 tablet_bootstrap.cc:492] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Bootstrap starting.
I20260809 15:02:02.971633 9111 tablet_bootstrap.cc:492] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Bootstrap starting.
I20260809 15:02:02.972226 9149 ts_tablet_manager.cc:1918] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:02.972435 9149 log.cc:1196] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/f5af290009ca46bd8a15a12d244248a1
I20260809 15:02:02.972757 9117 tablet_bootstrap.cc:654] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:02.973001 9149 ts_tablet_manager.cc:1939] T f5af290009ca46bd8a15a12d244248a1 P 6c4fada8d4024784aead7c57fa300b27: Deleting consensus metadata
I20260809 15:02:02.973024 9111 tablet_bootstrap.cc:654] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:02.975222 8220 catalog_manager.cc:4841] TS 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707): tablet f5af290009ca46bd8a15a12d244248a1 (table test-table [id=04a28bf5b95545579534d1585189438d]) successfully deleted
I20260809 15:02:02.979306 9117 tablet_bootstrap.cc:492] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: No bootstrap required, opened a new log
I20260809 15:02:02.979485 9117 ts_tablet_manager.cc:1397] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260809 15:02:02.980330 9114 tablet_bootstrap.cc:492] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Bootstrap starting.
I20260809 15:02:02.980356 9117 raft_consensus.cc:348] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.980636 9117 raft_consensus.cc:374] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:02.980756 9117 raft_consensus.cc:729] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Initialized, Role: FOLLOWER
I20260809 15:02:02.981060 9117 consensus_queue.cc:260] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.981735 9117 ts_tablet_manager.cc:1428] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 15:02:02.981806 9114 tablet_bootstrap.cc:654] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:02.986101 9150 tablet_replica.cc:330] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: stopping tablet replica
I20260809 15:02:02.986534 9150 raft_consensus.cc:2229] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:02:02.987149 9150 raft_consensus.cc:2258] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:02.988435 9111 tablet_bootstrap.cc:492] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: No bootstrap required, opened a new log
I20260809 15:02:02.988574 9111 ts_tablet_manager.cc:1397] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260809 15:02:02.989171 9111 raft_consensus.cc:348] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.989344 9111 raft_consensus.cc:374] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:02.989485 9111 raft_consensus.cc:729] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c710fa40b9b1496f92c4454a10d47e0f, State: Initialized, Role: FOLLOWER
I20260809 15:02:02.989707 9111 consensus_queue.cc:260] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.990249 9111 ts_tablet_manager.cc:1428] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 15:02:02.991566 9150 ts_tablet_manager.cc:1905] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:02.992692 9114 tablet_bootstrap.cc:492] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: No bootstrap required, opened a new log
I20260809 15:02:02.992830 9114 ts_tablet_manager.cc:1397] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260809 15:02:02.993561 9114 raft_consensus.cc:348] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.993772 9114 raft_consensus.cc:374] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:02.993870 9114 raft_consensus.cc:729] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4fada8d4024784aead7c57fa300b27, State: Initialized, Role: FOLLOWER
I20260809 15:02:02.994174 9114 consensus_queue.cc:260] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.996690 9114 ts_tablet_manager.cc:1428] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 15:02:02.997498 9128 raft_consensus.cc:482] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:02:02.997663 9128 raft_consensus.cc:504] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:02.998451 9150 ts_tablet_manager.cc:1918] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:02.998629 9150 log.cc:1196] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/f5af290009ca46bd8a15a12d244248a1
I20260809 15:02:02.999156 9150 ts_tablet_manager.cc:1939] T f5af290009ca46bd8a15a12d244248a1 P c8d591e1805f4250a81903db6c16a9c5: Deleting consensus metadata
I20260809 15:02:02.999395 9128 leader_election.cc:288] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871), c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821)
I20260809 15:02:03.001649 8223 catalog_manager.cc:4841] TS c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821): tablet f5af290009ca46bd8a15a12d244248a1 (table test-table [id=04a28bf5b95545579534d1585189438d]) successfully deleted
I20260809 15:02:03.015552 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.000384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:60401 (local address 127.27.186.130:39871)
0809 15:02:03.000550 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.000556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.008229 (+ 7673us) server_negotiation.cc:407] Connection header received
0809 15:02:03.008427 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.008433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.008515 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.008622 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:03.009538 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.010531 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.013032 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.014312 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.014437 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.014507 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.015137 (+ 630us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.015280 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:02:03.015584 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:02.999962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39871 (local address 127.27.186.129:60401)
0809 15:02:03.008113 (+ 8151us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:03.008138 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:03.008297 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:03.008976 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:03.008981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:03.009015 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:03.009377 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.009392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.010694 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.010700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.012850 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.012866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.014762 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.014768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.014963 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.014995 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:02:03.015089 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 15:02:03.015224 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8027,"thread_start_us":8780,"threads_started":1}
I20260809 15:02:03.016268 8759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef5f615967c4a128f510dc53ae95957" candidate_uuid: "6c4fada8d4024784aead7c57fa300b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f" is_pre_election: true
I20260809 15:02:03.016499 8759 raft_consensus.cc:2454] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c4fada8d4024784aead7c57fa300b27 in term 0.
I20260809 15:02:03.017105 8393 leader_election.cc:302] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4fada8d4024784aead7c57fa300b27, c710fa40b9b1496f92c4454a10d47e0f; no voters:
I20260809 15:02:03.017926 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.000993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:34855 (local address 127.27.186.131:33821)
0809 15:02:03.002712 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.002718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.008863 (+ 6145us) server_negotiation.cc:407] Connection header received
0809 15:02:03.009104 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.009108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.009174 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.009234 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:03.010577 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.011695 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.014360 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.015585 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.015745 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.015815 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.017524 (+ 1709us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.017651 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599}
I20260809 15:02:03.017912 9128 raft_consensus.cc:2781] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:02:03.018046 9128 raft_consensus.cc:482] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:02:03.018137 9128 raft_consensus.cc:3037] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.018755 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.000876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33821 (local address 127.27.186.129:34855)
0809 15:02:03.008770 (+ 7894us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:03.008788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:03.008960 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:03.010006 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:03.010010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:03.010065 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:03.010405 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.010417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.011823 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.011829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.014195 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.014211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.017122 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.017130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.017336 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.017370 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:02:03.018221 (+ 851us) client_negotiation.cc:240] Negotiation successful
0809 15:02:03.018368 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7778,"thread_start_us":6985,"threads_started":1}
I20260809 15:02:03.019595 8994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef5f615967c4a128f510dc53ae95957" candidate_uuid: "6c4fada8d4024784aead7c57fa300b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d591e1805f4250a81903db6c16a9c5" is_pre_election: true
I20260809 15:02:03.020018 8994 raft_consensus.cc:2454] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c4fada8d4024784aead7c57fa300b27 in term 0.
I20260809 15:02:03.021142 9128 raft_consensus.cc:504] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.022239 8759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef5f615967c4a128f510dc53ae95957" candidate_uuid: "6c4fada8d4024784aead7c57fa300b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f"
I20260809 15:02:03.022320 8994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef5f615967c4a128f510dc53ae95957" candidate_uuid: "6c4fada8d4024784aead7c57fa300b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d591e1805f4250a81903db6c16a9c5"
I20260809 15:02:03.022473 8759 raft_consensus.cc:3037] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.022512 8994 raft_consensus.cc:3037] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.025311 8759 raft_consensus.cc:2454] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c4fada8d4024784aead7c57fa300b27 in term 1.
I20260809 15:02:03.025646 8994 raft_consensus.cc:2454] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c4fada8d4024784aead7c57fa300b27 in term 1.
I20260809 15:02:03.026093 8394 leader_election.cc:302] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4fada8d4024784aead7c57fa300b27, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:03.026583 9128 leader_election.cc:288] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [CANDIDATE]: Term 1 election: Requested vote from peers c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871), c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821)
I20260809 15:02:03.026892 9160 raft_consensus.cc:2781] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:03.027091 9160 raft_consensus.cc:686] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 1 LEADER]: Becoming Leader. State: Replica: 6c4fada8d4024784aead7c57fa300b27, State: Running, Role: LEADER
I20260809 15:02:03.027503 9160 consensus_queue.cc:237] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.037195 8243 catalog_manager.cc:5495] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "6c4fada8d4024784aead7c57fa300b27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } health_report { overall_health: UNKNOWN } } }
Aug 09 15:02:03 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287721, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.190@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:03.096597 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.080628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43771 (local address 127.0.0.1:36864)
0809 15:02:03.081268 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:03.081286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:03.081416 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:03.081738 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:03.081743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:03.082718 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:03.083095 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.083107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.084530 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.084536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.086170 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.086185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.087319 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.087325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.087489 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.088223 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:03.088251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:03.090862 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:03.094377 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:03.094387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:03.094400 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:03.094783 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:03.095157 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:03.095160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:03.095162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:03.095286 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:03.095882 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:03.095887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:03.096026 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 15:02:03.096143 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 15:02:03.096309 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":343,"threads_started":1}
I20260809 15:02:03.096913 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.080532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.27.186.190:43771)
0809 15:02:03.080761 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.080767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.081369 (+ 602us) server_negotiation.cc:407] Connection header received
0809 15:02:03.081518 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.081523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.081600 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.081692 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:03.083244 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.084384 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.086311 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.086984 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.087074 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.091026 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:03.091033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:03.091038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:03.091074 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:03.094257 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:03.094903 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:03.094908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:03.094912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:03.095060 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:03.095393 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:03.095398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:03.095402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:03.095752 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:03.095912 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.096277 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.096508 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260809 15:02:03.101466 8243 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:36864:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 15:02:03.102505 8243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:02:03.108755 8472 tablet_service.cc:1467] Processing CreateTablet for tablet bab36cd66fbd4e18bb23d7a2096cceb1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fadef538e4a4e2d8d02a102613181e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:02:03.108845 8961 tablet_service.cc:1467] Processing CreateTablet for tablet bab36cd66fbd4e18bb23d7a2096cceb1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fadef538e4a4e2d8d02a102613181e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:02:03.109258 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab36cd66fbd4e18bb23d7a2096cceb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.109347 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab36cd66fbd4e18bb23d7a2096cceb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.112067 8721 tablet_service.cc:1467] Processing CreateTablet for tablet bab36cd66fbd4e18bb23d7a2096cceb1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fadef538e4a4e2d8d02a102613181e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 15:02:03.112506 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab36cd66fbd4e18bb23d7a2096cceb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.116222 9117 tablet_bootstrap.cc:492] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: Bootstrap starting.
I20260809 15:02:03.117941 9117 tablet_bootstrap.cc:654] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.122190 9111 tablet_bootstrap.cc:492] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: Bootstrap starting.
I20260809 15:02:03.122313 9114 tablet_bootstrap.cc:492] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: Bootstrap starting.
I20260809 15:02:03.123797 9111 tablet_bootstrap.cc:654] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.123970 9114 tablet_bootstrap.cc:654] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.127957 9111 tablet_bootstrap.cc:492] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: No bootstrap required, opened a new log
I20260809 15:02:03.128232 9111 ts_tablet_manager.cc:1397] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260809 15:02:03.128772 9114 tablet_bootstrap.cc:492] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: No bootstrap required, opened a new log
I20260809 15:02:03.128916 9114 ts_tablet_manager.cc:1397] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:02:03.128997 9111 raft_consensus.cc:348] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.129236 9111 raft_consensus.cc:374] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.129304 9111 raft_consensus.cc:729] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c710fa40b9b1496f92c4454a10d47e0f, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.129570 9114 raft_consensus.cc:348] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.129576 9111 consensus_queue.cc:260] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.129751 9114 raft_consensus.cc:374] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.129827 9114 raft_consensus.cc:729] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4fada8d4024784aead7c57fa300b27, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.130184 9111 ts_tablet_manager.cc:1428] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 15:02:03.130062 9114 consensus_queue.cc:260] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.130331 9126 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.130625 9114 ts_tablet_manager.cc:1428] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 15:02:03.130702 9126 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.130990 9128 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.131242 9128 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.132319 9117 tablet_bootstrap.cc:492] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: No bootstrap required, opened a new log
I20260809 15:02:03.132485 9117 ts_tablet_manager.cc:1397] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260809 15:02:03.133276 9117 raft_consensus.cc:348] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.133513 9117 raft_consensus.cc:374] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.133598 9117 raft_consensus.cc:729] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.133837 9117 consensus_queue.cc:260] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.134285 9127 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.134485 9127 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.135504 9117 ts_tablet_manager.cc:1428] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 15:02:03.375206 9127 raft_consensus.cc:482] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:02:03.375415 9127 raft_consensus.cc:504] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.376097 9127 leader_election.cc:288] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707), c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871)
I20260809 15:02:03.376679 8759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab36cd66fbd4e18bb23d7a2096cceb1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f" is_pre_election: true
I20260809 15:02:03.376665 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab36cd66fbd4e18bb23d7a2096cceb1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27" is_pre_election: true
I20260809 15:02:03.376950 8512 raft_consensus.cc:2454] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 0.
I20260809 15:02:03.376961 8759 raft_consensus.cc:2454] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 0.
I20260809 15:02:03.377414 8889 leader_election.cc:302] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:03.377702 9127 raft_consensus.cc:2781] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:02:03.377847 9127 raft_consensus.cc:482] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:02:03.377938 9127 raft_consensus.cc:3037] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.381052 9127 raft_consensus.cc:504] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.381686 9127 leader_election.cc:288] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Requested vote from peers 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707), c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871)
I20260809 15:02:03.382161 8759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab36cd66fbd4e18bb23d7a2096cceb1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f"
I20260809 15:02:03.382223 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab36cd66fbd4e18bb23d7a2096cceb1" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27"
I20260809 15:02:03.382411 8512 raft_consensus.cc:3037] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.382422 8759 raft_consensus.cc:3037] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.384368 8512 raft_consensus.cc:2454] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 1.
I20260809 15:02:03.384581 8759 raft_consensus.cc:2454] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 1.
I20260809 15:02:03.384758 8887 leader_election.cc:302] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4fada8d4024784aead7c57fa300b27, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:03.385044 9127 raft_consensus.cc:2781] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:03.385179 9127 raft_consensus.cc:686] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [term 1 LEADER]: Becoming Leader. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Running, Role: LEADER
I20260809 15:02:03.385444 9127 consensus_queue.cc:237] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.386925 9131 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: TxnStatusTablet state changed. Reason: New leader c8d591e1805f4250a81903db6c16a9c5. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.387188 9131 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:02:03.387802 9180 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 15:02:03.388005 9180 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 15:02:03.388360 8243 catalog_manager.cc:5495] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } health_report { overall_health: HEALTHY } } }
I20260809 15:02:03.411531 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.27.186.131:33821)
0809 15:02:03.406162 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.406168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.406194 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:02:03.406412 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.406418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.406501 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.406607 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:03.406947 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.408076 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.409446 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.410208 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.410359 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.410837 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.410940 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.411014 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.411158 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.411241 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 15:02:03.411550 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.405859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33821 (local address 127.0.0.1:34114)
0809 15:02:03.406107 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:03.406124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:03.406238 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:03.406639 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:03.406644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:03.406659 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:02:03.406819 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.406827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.408196 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.408201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.409306 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.409320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.410389 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.410395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.410606 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.410662 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.411036 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.411057 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 15:02:03.411126 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 15:02:03.411273 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 15:02:03.411978 9128 consensus_queue.cc:1035] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:03.416118 8759 raft_consensus.cc:1261] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Refusing update from remote peer c8d591e1805f4250a81903db6c16a9c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:03.416812 8512 raft_consensus.cc:1261] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Refusing update from remote peer c8d591e1805f4250a81903db6c16a9c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:03.417529 9127 consensus_queue.cc:1035] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:03.418090 9127 consensus_queue.cc:1035] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:03.419972 9128 consensus_queue.cc:1035] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:03.429759 9126 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: TxnStatusTablet state changed. Reason: New leader c8d591e1805f4250a81903db6c16a9c5. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.430059 9126 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.432672 9131 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.432973 9131 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:02:03.434620 9128 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: TxnStatusTablet state changed. Reason: New leader c8d591e1805f4250a81903db6c16a9c5. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.434880 9128 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.436229 9127 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.436486 9127 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c8d591e1805f4250a81903db6c16a9c5: This TxnStatusTablet replica's current role is: LEADER
I20260809 15:02:03.437072 9128 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.438154 9126 tablet_replica.cc:439] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } } }
I20260809 15:02:03.438400 9126 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P c710fa40b9b1496f92c4454a10d47e0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.438555 9128 tablet_replica.cc:442] T bab36cd66fbd4e18bb23d7a2096cceb1 P 6c4fada8d4024784aead7c57fa300b27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 15:02:03.458999 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.447489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.27.186.129:35707)
0809 15:02:03.453906 (+ 6417us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.453911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.453928 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:02:03.453999 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.454002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.454131 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.454274 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:03.454754 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.455546 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.456862 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.457262 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.457379 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.457809 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.458383 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.458526 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.458619 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.458699 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6306,"thread_start_us":6198,"threads_started":1}
I20260809 15:02:03.474205 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.468316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:42533 (local address 127.27.186.131:33821)
0809 15:02:03.468466 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.468470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.468984 (+ 514us) server_negotiation.cc:407] Connection header received
0809 15:02:03.469113 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.469119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.469205 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.469297 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:03.469684 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.470836 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.472208 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.472916 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.473047 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.473451 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.473598 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.473690 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.473803 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.473880 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:02:03.474220 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.468347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33821 (local address 127.27.186.129:42533)
0809 15:02:03.468888 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:03.468903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:03.469008 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:03.469306 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:03.469309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:03.469318 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:02:03.469552 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.469559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.470948 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.470951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.472052 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.472064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.473110 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.473116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.473238 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.473290 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.473688 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.473697 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 15:02:03.473761 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 15:02:03.473852 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
I20260809 15:02:03.509546 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.503102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:34691 (local address 127.27.186.129:35707)
0809 15:02:03.503242 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.503247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.503879 (+ 632us) server_negotiation.cc:407] Connection header received
0809 15:02:03.504052 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.504058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.504128 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.504199 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:03.504773 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.505858 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.507494 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.508130 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.508254 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.508716 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.508857 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.508943 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.509219 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.509287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 15:02:03.509725 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.503184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:35707 (local address 127.27.186.131:34691)
0809 15:02:03.503787 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:03.503803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:03.503936 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:03.504231 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:03.504235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:03.504247 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:02:03.504591 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.504602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.506002 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.506005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.507355 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:03.507368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.508300 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.508307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:03.508466 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.508535 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.509081 (+ 546us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.509091 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 15:02:03.509161 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 15:02:03.509287 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
I20260809 15:02:03.546847 8240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:03.547086 8240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:03.550292 8240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Sending DeleteTablet for 3 replicas of tablet 7ef5f615967c4a128f510dc53ae95957
I20260809 15:02:03.550987 8472 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef5f615967c4a128f510dc53ae95957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45576
I20260809 15:02:03.551371 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef5f615967c4a128f510dc53ae95957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:41024
I20260809 15:02:03.552047 8961 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef5f615967c4a128f510dc53ae95957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34100
I20260809 15:02:03.577267 9212 tablet_replica.cc:330] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: stopping tablet replica
I20260809 15:02:03.577474 9212 raft_consensus.cc:2229] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:03.577692 9212 raft_consensus.cc:2258] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:03.578475 9213 tablet_replica.cc:330] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: stopping tablet replica
I20260809 15:02:03.578879 9212 ts_tablet_manager.cc:1905] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:03.580335 9213 raft_consensus.cc:2229] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:03.580929 9213 raft_consensus.cc:2258] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:03.582589 9213 ts_tablet_manager.cc:1905] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:03.585827 9212 ts_tablet_manager.cc:1918] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:02:03.585959 9212 log.cc:1196] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/7ef5f615967c4a128f510dc53ae95957
I20260809 15:02:03.586444 9212 ts_tablet_manager.cc:1939] T 7ef5f615967c4a128f510dc53ae95957 P c710fa40b9b1496f92c4454a10d47e0f: Deleting consensus metadata
I20260809 15:02:03.586859 9211 tablet_replica.cc:330] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: stopping tablet replica
I20260809 15:02:03.587445 8222 catalog_manager.cc:4841] TS c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871): tablet 7ef5f615967c4a128f510dc53ae95957 (table test-table [id=09a9fd4c1df8478b98a25260ea4afdd1]) successfully deleted
I20260809 15:02:03.587783 9211 raft_consensus.cc:2229] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:02:03.588377 9211 raft_consensus.cc:2258] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:03.596035 9211 ts_tablet_manager.cc:1905] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:03.603772 9211 ts_tablet_manager.cc:1918] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:02:03.604547 9213 ts_tablet_manager.cc:1918] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:02:03.604753 9211 log.cc:1196] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/7ef5f615967c4a128f510dc53ae95957
I20260809 15:02:03.605278 9211 ts_tablet_manager.cc:1939] T 7ef5f615967c4a128f510dc53ae95957 P 6c4fada8d4024784aead7c57fa300b27: Deleting consensus metadata
I20260809 15:02:03.604774 9213 log.cc:1196] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/7ef5f615967c4a128f510dc53ae95957
I20260809 15:02:03.605947 9213 ts_tablet_manager.cc:1939] T 7ef5f615967c4a128f510dc53ae95957 P c8d591e1805f4250a81903db6c16a9c5: Deleting consensus metadata
I20260809 15:02:03.606339 8220 catalog_manager.cc:4841] TS 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707): tablet 7ef5f615967c4a128f510dc53ae95957 (table test-table [id=09a9fd4c1df8478b98a25260ea4afdd1]) successfully deleted
I20260809 15:02:03.607537 8223 catalog_manager.cc:4841] TS c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821): tablet 7ef5f615967c4a128f510dc53ae95957 (table test-table [id=09a9fd4c1df8478b98a25260ea4afdd1]) successfully deleted
Aug 09 15:02:03 dist-test-slave-q9h9 krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 15:02:03 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:03 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
W20260809 15:02:03.673453 9225 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:03.643262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.27.186.190:43771)
0809 15:02:03.660866 (+ 17604us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.660873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.660894 (+ 21us) server_negotiation.cc:407] Connection header received
0809 15:02:03.660958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.660963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.661043 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.661154 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:03.663232 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.664375 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.666220 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.666862 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.666985 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.673062 (+ 6077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36870: BlockingRecv error: recv got EOF from 127.0.0.1:36870 (error 108)
Metrics: {"server-negotiator.queue_time_us":17475,"thread_start_us":19789,"threads_started":1}
Aug 09 15:02:03 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM
I20260809 15:02:03.747923 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.720246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.27.186.190:43771)
0809 15:02:03.720432 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.720439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.726239 (+ 5800us) server_negotiation.cc:407] Connection header received
0809 15:02:03.726433 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.726439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.726535 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.726639 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:03.728553 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.729650 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.731579 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.732217 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.732338 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.741536 (+ 9198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:03.741543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:03.741549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:03.741590 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:03.745023 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:03.745807 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:03.745812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:03.745816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:03.745977 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:03.746426 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:03.746431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:03.746435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:03.746765 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:03.747015 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.747373 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.747593 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:02:03.751787 8240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:02:03.752568 8240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:02:03.760540 8472 tablet_service.cc:1467] Processing CreateTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe (DEFAULT_TABLE table=test-table [id=f6e18bc9999e40078182cca80e52cce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:03.760962 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.761104 8721 tablet_service.cc:1467] Processing CreateTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe (DEFAULT_TABLE table=test-table [id=f6e18bc9999e40078182cca80e52cce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:03.761566 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.767015 8961 tablet_service.cc:1467] Processing CreateTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe (DEFAULT_TABLE table=test-table [id=f6e18bc9999e40078182cca80e52cce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:03.767469 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.791301 9238 tablet_bootstrap.cc:492] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Bootstrap starting.
I20260809 15:02:03.794557 9237 tablet_bootstrap.cc:492] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Bootstrap starting.
I20260809 15:02:03.796327 9237 tablet_bootstrap.cc:654] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.799440 9238 tablet_bootstrap.cc:654] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.800992 9236 tablet_bootstrap.cc:492] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Bootstrap starting.
I20260809 15:02:03.801477 9237 tablet_bootstrap.cc:492] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: No bootstrap required, opened a new log
I20260809 15:02:03.801628 9237 ts_tablet_manager.cc:1397] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260809 15:02:03.802425 9237 raft_consensus.cc:348] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.802682 9237 raft_consensus.cc:374] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.802786 9237 raft_consensus.cc:729] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c710fa40b9b1496f92c4454a10d47e0f, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.803058 9237 consensus_queue.cc:260] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.803377 9236 tablet_bootstrap.cc:654] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.805945 9237 ts_tablet_manager.cc:1428] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 15:02:03.807350 9238 tablet_bootstrap.cc:492] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: No bootstrap required, opened a new log
I20260809 15:02:03.807518 9238 ts_tablet_manager.cc:1397] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260809 15:02:03.808382 9238 raft_consensus.cc:348] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.808642 9238 raft_consensus.cc:374] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.808763 9238 raft_consensus.cc:729] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.809058 9238 consensus_queue.cc:260] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.813210 9238 ts_tablet_manager.cc:1428] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 15:02:03.815429 9127 raft_consensus.cc:482] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:02:03.815593 9127 raft_consensus.cc:504] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.817137 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ccdcd2b8bd4f81a57d12c1a09202fe" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27" is_pre_election: true
W20260809 15:02:03.818097 8887 leader_election.cc:341] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:02:03.818436 9127 leader_election.cc:288] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707), c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871)
I20260809 15:02:03.819082 8758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ccdcd2b8bd4f81a57d12c1a09202fe" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f" is_pre_election: true
I20260809 15:02:03.819316 8758 raft_consensus.cc:2454] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 0.
I20260809 15:02:03.819787 8889 leader_election.cc:302] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters: 6c4fada8d4024784aead7c57fa300b27
I20260809 15:02:03.820278 9127 raft_consensus.cc:2781] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:02:03.820384 9127 raft_consensus.cc:482] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:02:03.820498 9127 raft_consensus.cc:3037] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.824651 9127 raft_consensus.cc:504] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.825564 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ccdcd2b8bd4f81a57d12c1a09202fe" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27"
W20260809 15:02:03.826089 8887 leader_election.cc:341] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:02:03.826694 8758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ccdcd2b8bd4f81a57d12c1a09202fe" candidate_uuid: "c8d591e1805f4250a81903db6c16a9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c710fa40b9b1496f92c4454a10d47e0f"
I20260809 15:02:03.826895 8758 raft_consensus.cc:3037] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.829640 8758 raft_consensus.cc:2454] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d591e1805f4250a81903db6c16a9c5 in term 1.
I20260809 15:02:03.830111 8889 leader_election.cc:302] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters: 6c4fada8d4024784aead7c57fa300b27
I20260809 15:02:03.830370 9131 raft_consensus.cc:2781] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:03.833563 9127 leader_election.cc:288] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [CANDIDATE]: Term 1 election: Requested vote from peers 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707), c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871)
I20260809 15:02:03.841261 9131 raft_consensus.cc:686] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 1 LEADER]: Becoming Leader. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Running, Role: LEADER
I20260809 15:02:03.841575 9131 consensus_queue.cc:237] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.849332 8245 catalog_manager.cc:5495] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "c8d591e1805f4250a81903db6c16a9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } health_report { overall_health: HEALTHY } } }
I20260809 15:02:03.854748 9236 tablet_bootstrap.cc:492] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: No bootstrap required, opened a new log
I20260809 15:02:03.854957 9236 ts_tablet_manager.cc:1397] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Time spent bootstrapping tablet: real 0.054s user 0.019s sys 0.027s
I20260809 15:02:03.855866 9236 raft_consensus.cc:348] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.856133 9236 raft_consensus.cc:374] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.856238 9236 raft_consensus.cc:729] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4fada8d4024784aead7c57fa300b27, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.856495 9236 consensus_queue.cc:260] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.857076 9236 ts_tablet_manager.cc:1428] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 15:02:03.873140 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:03.866877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.27.186.131:33821)
0809 15:02:03.867052 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:03.867058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:03.867234 (+ 176us) server_negotiation.cc:407] Connection header received
0809 15:02:03.867402 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:03.867407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:03.867510 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:03.867603 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:03.868002 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.869142 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.871073 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:03.871720 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:03.871844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:03.872327 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:03.872449 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:03.872535 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:03.872667 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 15:02:03.872756 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:02:03.875835 8512 raft_consensus.cc:3037] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:03.878306 8512 raft_consensus.cc:1261] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Refusing update from remote peer c8d591e1805f4250a81903db6c16a9c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:03.878440 8758 raft_consensus.cc:1261] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Refusing update from remote peer c8d591e1805f4250a81903db6c16a9c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:03.878868 9246 consensus_queue.cc:1035] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:03.879760 9246 consensus_queue.cc:1035] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:03.891125 8245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:03.891358 8245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:03.894510 8245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Sending DeleteTablet for 3 replicas of tablet 41ccdcd2b8bd4f81a57d12c1a09202fe
I20260809 15:02:03.895133 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:41024
I20260809 15:02:03.895188 8472 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45576
I20260809 15:02:03.895430 9212 tablet_replica.cc:330] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: stopping tablet replica
I20260809 15:02:03.895452 9211 tablet_replica.cc:330] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: stopping tablet replica
I20260809 15:02:03.895628 9212 raft_consensus.cc:2229] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:03.895644 9211 raft_consensus.cc:2229] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:03.895843 9211 raft_consensus.cc:2258] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:03.895838 9212 raft_consensus.cc:2258] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:03.896484 8961 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34100
I20260809 15:02:03.896733 9213 tablet_replica.cc:330] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: stopping tablet replica
I20260809 15:02:03.896919 9213 raft_consensus.cc:2229] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:02:03.897241 9212 ts_tablet_manager.cc:1905] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:03.897205 9213 raft_consensus.cc:2258] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:03.897462 8245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:02:03.898239 8245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:02:03.898280 9213 ts_tablet_manager.cc:1905] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:03.901319 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:03 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:41024
I20260809 15:02:03.901754 9211 ts_tablet_manager.cc:1905] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:03.904428 9212 ts_tablet_manager.cc:1918] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:03.904561 9212 log.cc:1196] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/41ccdcd2b8bd4f81a57d12c1a09202fe
I20260809 15:02:03.904686 9213 ts_tablet_manager.cc:1918] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:03.904808 9213 log.cc:1196] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/41ccdcd2b8bd4f81a57d12c1a09202fe
I20260809 15:02:03.904978 9212 ts_tablet_manager.cc:1939] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c710fa40b9b1496f92c4454a10d47e0f: Deleting consensus metadata
I20260809 15:02:03.905191 9213 ts_tablet_manager.cc:1939] T 41ccdcd2b8bd4f81a57d12c1a09202fe P c8d591e1805f4250a81903db6c16a9c5: Deleting consensus metadata
I20260809 15:02:03.905912 8222 catalog_manager.cc:4841] TS c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871): tablet 41ccdcd2b8bd4f81a57d12c1a09202fe (table test-table [id=f6e18bc9999e40078182cca80e52cce3]) successfully deleted
I20260809 15:02:03.906113 8223 catalog_manager.cc:4841] TS c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821): tablet 41ccdcd2b8bd4f81a57d12c1a09202fe (table test-table [id=f6e18bc9999e40078182cca80e52cce3]) successfully deleted
W20260809 15:02:03.908844 8222 catalog_manager.cc:4808] TS c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871): delete failed for tablet 41ccdcd2b8bd4f81a57d12c1a09202fe because the tablet was not found. No further retry: Not found: Tablet not found: 41ccdcd2b8bd4f81a57d12c1a09202fe
I20260809 15:02:03.909493 9211 ts_tablet_manager.cc:1918] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:03.909752 9211 log.cc:1196] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/41ccdcd2b8bd4f81a57d12c1a09202fe
I20260809 15:02:03.910198 8721 tablet_service.cc:1467] Processing CreateTablet for tablet 08eec5f88d5443e7b172b6914e905911 (DEFAULT_TABLE table=test-table [id=f6d91ff8ebdd46ad910a6b2ca493257e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:03.910312 9211 ts_tablet_manager.cc:1939] T 41ccdcd2b8bd4f81a57d12c1a09202fe P 6c4fada8d4024784aead7c57fa300b27: Deleting consensus metadata
I20260809 15:02:03.910490 8472 tablet_service.cc:1467] Processing CreateTablet for tablet 08eec5f88d5443e7b172b6914e905911 (DEFAULT_TABLE table=test-table [id=f6d91ff8ebdd46ad910a6b2ca493257e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:03.910591 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08eec5f88d5443e7b172b6914e905911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.910897 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08eec5f88d5443e7b172b6914e905911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.912372 8961 tablet_service.cc:1467] Processing CreateTablet for tablet 08eec5f88d5443e7b172b6914e905911 (DEFAULT_TABLE table=test-table [id=f6d91ff8ebdd46ad910a6b2ca493257e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:03.912762 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08eec5f88d5443e7b172b6914e905911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:03.916373 9237 tablet_bootstrap.cc:492] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Bootstrap starting.
I20260809 15:02:03.917114 9236 tablet_bootstrap.cc:492] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Bootstrap starting.
I20260809 15:02:03.917994 9237 tablet_bootstrap.cc:654] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.918514 8220 catalog_manager.cc:4841] TS 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707): tablet 41ccdcd2b8bd4f81a57d12c1a09202fe (table test-table [id=f6e18bc9999e40078182cca80e52cce3]) successfully deleted
I20260809 15:02:03.918608 9236 tablet_bootstrap.cc:654] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.924007 9238 tablet_bootstrap.cc:492] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Bootstrap starting.
I20260809 15:02:03.925601 9238 tablet_bootstrap.cc:654] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:03.928747 9238 tablet_bootstrap.cc:492] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: No bootstrap required, opened a new log
I20260809 15:02:03.929003 9238 ts_tablet_manager.cc:1397] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 15:02:03.929790 9238 raft_consensus.cc:348] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.930320 9237 tablet_bootstrap.cc:492] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: No bootstrap required, opened a new log
I20260809 15:02:03.930480 9237 ts_tablet_manager.cc:1397] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260809 15:02:03.931243 9238 raft_consensus.cc:374] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.931196 9237 raft_consensus.cc:348] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.931443 9237 raft_consensus.cc:374] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.931443 9238 raft_consensus.cc:729] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d591e1805f4250a81903db6c16a9c5, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.931537 9237 raft_consensus.cc:729] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c710fa40b9b1496f92c4454a10d47e0f, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.931836 9238 consensus_queue.cc:260] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.932050 9237 consensus_queue.cc:260] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.933323 9237 ts_tablet_manager.cc:1428] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 15:02:03.934899 9236 tablet_bootstrap.cc:492] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: No bootstrap required, opened a new log
I20260809 15:02:03.935093 9236 ts_tablet_manager.cc:1397] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20260809 15:02:03.935367 9238 ts_tablet_manager.cc:1428] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260809 15:02:03.935911 9236 raft_consensus.cc:348] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.936156 9236 raft_consensus.cc:374] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:03.936260 9236 raft_consensus.cc:729] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4fada8d4024784aead7c57fa300b27, State: Initialized, Role: FOLLOWER
I20260809 15:02:03.936527 9236 consensus_queue.cc:260] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:03.937656 9236 ts_tablet_manager.cc:1428] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 15:02:04.057322 9126 raft_consensus.cc:482] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:02:04.057569 9126 raft_consensus.cc:504] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:04.059381 9126 leader_election.cc:288] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707), c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821)
I20260809 15:02:04.067322 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.059747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:55699 (local address 127.27.186.131:33821)
0809 15:02:04.059914 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.059920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.060520 (+ 600us) server_negotiation.cc:407] Connection header received
0809 15:02:04.060772 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.060778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.060850 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.060906 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:04.061633 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.062647 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.065016 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.066235 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.066353 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.066425 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.066929 (+ 504us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.067061 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:02:04.068179 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.059619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33821 (local address 127.27.186.130:55699)
0809 15:02:04.060418 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:04.060437 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:04.060628 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:04.061088 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:04.061093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:04.061118 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:04.061455 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.061466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.062840 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.062845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.064842 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.064858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.066569 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.066575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.066748 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.066782 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:02:04.067707 (+ 925us) client_negotiation.cc:240] Negotiation successful
0809 15:02:04.067840 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":570,"threads_started":1}
I20260809 15:02:04.068989 8994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eec5f88d5443e7b172b6914e905911" candidate_uuid: "c710fa40b9b1496f92c4454a10d47e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d591e1805f4250a81903db6c16a9c5" is_pre_election: true
I20260809 15:02:04.069303 8994 raft_consensus.cc:2454] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c710fa40b9b1496f92c4454a10d47e0f in term 0.
I20260809 15:02:04.069900 8628 leader_election.cc:302] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:04.070389 9126 raft_consensus.cc:2781] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:02:04.070519 9126 raft_consensus.cc:482] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:02:04.070616 9126 raft_consensus.cc:3037] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:04.074170 9126 raft_consensus.cc:504] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:04.075495 8994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eec5f88d5443e7b172b6914e905911" candidate_uuid: "c710fa40b9b1496f92c4454a10d47e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d591e1805f4250a81903db6c16a9c5"
I20260809 15:02:04.075733 8994 raft_consensus.cc:3037] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:04.076584 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.060052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:44405 (local address 127.27.186.129:35707)
0809 15:02:04.069148 (+ 9096us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.069154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.069172 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:02:04.069337 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.069342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.069481 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.069945 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:04.070282 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.071543 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.073825 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.075137 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.076055 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.076124 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.076170 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.076301 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8952,"threads_started":1}
I20260809 15:02:04.078743 8994 raft_consensus.cc:2454] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c710fa40b9b1496f92c4454a10d47e0f in term 1.
I20260809 15:02:04.074925 9126 leader_election.cc:288] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [CANDIDATE]: Term 1 election: Requested vote from peers 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707), c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821)
I20260809 15:02:04.079631 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.059709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:35707 (local address 127.27.186.130:44405)
0809 15:02:04.068594 (+ 8885us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:04.068611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:04.068700 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:04.069615 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:04.069620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:04.069644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:04.069811 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.069822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.071690 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.071696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.073573 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.073588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.075340 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.075346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.075562 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.075595 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:02:04.075723 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 15:02:04.075896 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8796,"threads_started":1}
I20260809 15:02:04.079185 8628 leader_election.cc:302] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c710fa40b9b1496f92c4454a10d47e0f, c8d591e1805f4250a81903db6c16a9c5; no voters:
I20260809 15:02:04.080011 9126 raft_consensus.cc:2781] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:04.082032 9126 raft_consensus.cc:686] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 1 LEADER]: Becoming Leader. State: Replica: c710fa40b9b1496f92c4454a10d47e0f, State: Running, Role: LEADER
I20260809 15:02:04.082484 9126 consensus_queue.cc:237] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } }
I20260809 15:02:04.086071 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eec5f88d5443e7b172b6914e905911" candidate_uuid: "c710fa40b9b1496f92c4454a10d47e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27" is_pre_election: true
I20260809 15:02:04.086340 8512 raft_consensus.cc:2454] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c710fa40b9b1496f92c4454a10d47e0f in term 0.
I20260809 15:02:04.086930 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eec5f88d5443e7b172b6914e905911" candidate_uuid: "c710fa40b9b1496f92c4454a10d47e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fada8d4024784aead7c57fa300b27"
I20260809 15:02:04.087126 8512 raft_consensus.cc:3037] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:04.090382 8512 raft_consensus.cc:2454] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c710fa40b9b1496f92c4454a10d47e0f in term 1.
I20260809 15:02:04.092185 8240 catalog_manager.cc:5495] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130). New cstate: current_term: 1 leader_uuid: "c710fa40b9b1496f92c4454a10d47e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c710fa40b9b1496f92c4454a10d47e0f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 } health_report { overall_health: UNKNOWN } } }
I20260809 15:02:04.125106 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.119633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.27.186.130:39871)
0809 15:02:04.120192 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.120196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.120212 (+ 16us) server_negotiation.cc:407] Connection header received
0809 15:02:04.120291 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.120294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.120364 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.120495 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:04.120903 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.121644 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.122957 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.123365 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.123472 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.123918 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:04.124496 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:04.124633 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.124732 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.124818 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":326,"threads_started":1}
I20260809 15:02:04.138113 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.133708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:34051 (local address 127.27.186.131:33821)
0809 15:02:04.133861 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.133865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.134322 (+ 457us) server_negotiation.cc:407] Connection header received
0809 15:02:04.134475 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.134479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.134541 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.134601 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:04.134996 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.135637 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.136844 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.137194 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.137282 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.137608 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:04.137692 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:04.137750 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.137831 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.137889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:02:04.138213 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.133594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33821 (local address 127.27.186.130:34051)
0809 15:02:04.134240 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:04.134255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:04.134372 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:04.134614 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:04.134616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:04.134625 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:02:04.134886 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.134893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.135729 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.135732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.136739 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.136750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.137332 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.137336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.137444 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.137490 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:02:04.137755 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:02:04.137761 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:02:04.137812 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 15:02:04.137899 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":304,"threads_started":1}
I20260809 15:02:04.145603 8512 raft_consensus.cc:1261] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Refusing update from remote peer c710fa40b9b1496f92c4454a10d47e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:04.145642 8994 raft_consensus.cc:1261] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Refusing update from remote peer c710fa40b9b1496f92c4454a10d47e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:04.146219 9126 consensus_queue.cc:1035] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4fada8d4024784aead7c57fa300b27" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:04.146483 9259 consensus_queue.cc:1035] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d591e1805f4250a81903db6c16a9c5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:04.178306 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.172412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:48019 (local address 127.27.186.130:39871)
0809 15:02:04.172587 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.172592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.173097 (+ 505us) server_negotiation.cc:407] Connection header received
0809 15:02:04.173224 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.173229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.173321 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.173400 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:04.173792 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.174982 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.176573 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.177241 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.177352 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.177634 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:04.177754 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:04.177826 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.177943 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.178046 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:02:04.178385 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.172413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39871 (local address 127.27.186.131:48019)
0809 15:02:04.173008 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:04.173026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:04.173129 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:04.173438 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:04.173443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:04.173456 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 15:02:04.173629 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.173636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.175114 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.175119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.176447 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:04.176460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.177355 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.177359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:04.177481 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.177514 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 15:02:04.177828 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 15:02:04.177834 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:02:04.177896 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 15:02:04.178045 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":324,"threads_started":1}
I20260809 15:02:04.214564 8240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:04.214767 8240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:04.217463 8240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02f0ba040f354a2d962cb1e98c859d2c: Sending DeleteTablet for 3 replicas of tablet 08eec5f88d5443e7b172b6914e905911
I20260809 15:02:04.218165 8472 tablet_service.cc:1514] Processing DeleteTablet for tablet 08eec5f88d5443e7b172b6914e905911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:04 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45576
I20260809 15:02:04.218425 9211 tablet_replica.cc:330] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: stopping tablet replica
I20260809 15:02:04.218586 9211 raft_consensus.cc:2229] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:04.218783 9211 raft_consensus.cc:2258] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:04.219273 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet 08eec5f88d5443e7b172b6914e905911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:04 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:41024
I20260809 15:02:04.219431 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 8364
I20260809 15:02:04.219539 9212 tablet_replica.cc:330] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: stopping tablet replica
I20260809 15:02:04.219673 8961 tablet_service.cc:1514] Processing DeleteTablet for tablet 08eec5f88d5443e7b172b6914e905911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:04 UTC) from {username='oryx', principal='oryx/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34100
I20260809 15:02:04.219713 9212 raft_consensus.cc:2229] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:02:04.219947 9213 tablet_replica.cc:330] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: stopping tablet replica
I20260809 15:02:04.220108 9213 raft_consensus.cc:2229] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:04.220180 9212 raft_consensus.cc:2258] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:04.220283 9213 raft_consensus.cc:2258] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:04.221117 9211 ts_tablet_manager.cc:1905] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:04.221460 9212 ts_tablet_manager.cc:1905] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:04.226919 9213 ts_tablet_manager.cc:1905] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:04.227818 9211 ts_tablet_manager.cc:1918] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:02:04.227952 9211 log.cc:1196] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/08eec5f88d5443e7b172b6914e905911
I20260809 15:02:04.228335 9211 ts_tablet_manager.cc:1939] T 08eec5f88d5443e7b172b6914e905911 P 6c4fada8d4024784aead7c57fa300b27: Deleting consensus metadata
I20260809 15:02:04.229351 8220 catalog_manager.cc:4841] TS 6c4fada8d4024784aead7c57fa300b27 (127.27.186.129:35707): tablet 08eec5f88d5443e7b172b6914e905911 (table test-table [id=f6d91ff8ebdd46ad910a6b2ca493257e]) successfully deleted
I20260809 15:02:04.233178 9213 ts_tablet_manager.cc:1918] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:02:04.233310 9213 log.cc:1196] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/08eec5f88d5443e7b172b6914e905911
I20260809 15:02:04.233700 9213 ts_tablet_manager.cc:1939] T 08eec5f88d5443e7b172b6914e905911 P c8d591e1805f4250a81903db6c16a9c5: Deleting consensus metadata
I20260809 15:02:04.235390 8223 catalog_manager.cc:4841] TS c8d591e1805f4250a81903db6c16a9c5 (127.27.186.131:33821): tablet 08eec5f88d5443e7b172b6914e905911 (table test-table [id=f6d91ff8ebdd46ad910a6b2ca493257e]) successfully deleted
I20260809 15:02:04.236356 9212 ts_tablet_manager.cc:1918] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 15:02:04.236485 9212 log.cc:1196] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/08eec5f88d5443e7b172b6914e905911
I20260809 15:02:04.236866 9212 ts_tablet_manager.cc:1939] T 08eec5f88d5443e7b172b6914e905911 P c710fa40b9b1496f92c4454a10d47e0f: Deleting consensus metadata
I20260809 15:02:04.237835 8222 catalog_manager.cc:4841] TS c710fa40b9b1496f92c4454a10d47e0f (127.27.186.130:39871): tablet 08eec5f88d5443e7b172b6914e905911 (table test-table [id=f6d91ff8ebdd46ad910a6b2ca493257e]) successfully deleted
Aug 09 15:02:04 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.129@KRBTEST.COM
I20260809 15:02:04.246591 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.221128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.27.186.129:35707)
0809 15:02:04.226204 (+ 5076us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.226210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.230234 (+ 4024us) server_negotiation.cc:407] Connection header received
0809 15:02:04.230450 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.230456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.230538 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.230649 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:04.232177 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.233340 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.235257 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.235896 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.236027 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.241167 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:04.241181 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:04.241197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:04.241227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:04.243892 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:04.244635 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:04.244641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:04.244644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:04.244765 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:04.245173 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:04.245178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:04.245182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:04.245572 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:04.245805 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.246175 (+ 370us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.246346 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4913}
I20260809 15:02:04.247232 8582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:04.366707 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 8364
I20260809 15:02:04.407783 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 8598
Aug 09 15:02:04 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.130@KRBTEST.COM
I20260809 15:02:04.422986 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.408528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.27.186.130:39871)
0809 15:02:04.408734 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.408739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.408774 (+ 35us) server_negotiation.cc:407] Connection header received
0809 15:02:04.408955 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.408961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.409057 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.409149 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:04.410409 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.411136 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.412386 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.412829 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.412945 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.417022 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:04.417033 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:04.417046 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:04.417085 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:04.420226 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:04.420857 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:04.420864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:04.420866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:04.420996 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:04.421350 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:04.421354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:04.421356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:04.421694 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:04.422050 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.422428 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.422667 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 15:02:04.423635 8839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:04.542755 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 8598
I20260809 15:02:04.568280 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 8862
Aug 09 15:02:04 dist-test-slave-q9h9 krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.186.131@KRBTEST.COM
I20260809 15:02:04.582834 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:04.569022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.27.186.131:33821)
0809 15:02:04.569218 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:04.569223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:04.569243 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:02:04.569432 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:04.569438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:04.569512 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:04.569595 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:04.570940 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.571633 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.572873 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:04.573328 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:04.573437 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:04.577535 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:04.577550 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:04.577567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:04.577598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:04.580240 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:04.580848 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:04.580855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:04.580858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:04.580985 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:04.581326 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:04.581330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:04.581332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:04.581663 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:04.581957 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:04.582282 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 15:02:04.582535 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:02:04.583425 9068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:04.722954 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 8862
I20260809 15:02:04.757123 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 8188
I20260809 15:02:04.758215 8315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:04.870289 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 8188
2026-08-09T15:02:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5591 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:02:04 dist-test-slave-q9h9 krb5kdc[9293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:02:04 dist-test-slave-q9h9 krb5kdc[9293](info): set up 2 sockets
Aug 09 15:02:04 dist-test-slave-q9h9 krb5kdc[9293](info): commencing operation
krb5kdc: starting...
W20260809 15:02:06.086776 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:02:06 dist-test-slave-q9h9 krb5kdc[9293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:02:06.294610 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37617
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:44587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:37617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:06.598559 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:06.598821 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:06.598937 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:06.605669 9320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:06.605770 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:06.605854 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:06.605934 9320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:06.606001 9320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:06.615075 9320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:44587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:37617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:06.616138 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:06.617282 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:06.628011 9328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:06.628031 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:06.628089 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:06.628631 9320 server_base.cc:1034] running on GCE node
I20260809 15:02:06.629269 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:06.630503 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:06.631692 9320 hybrid_clock.cc:648] HybridClock initialized: now 1786287726631654 us; error 28 us; skew 500 ppm
I20260809 15:02:06.631992 9320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T15:02:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1821 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:02:06 dist-test-slave-q9h9 krb5kdc[9337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:02:06 dist-test-slave-q9h9 krb5kdc[9337](info): set up 2 sockets
Aug 09 15:02:06 dist-test-slave-q9h9 krb5kdc[9337](info): commencing operation
krb5kdc: starting...
W20260809 15:02:07.891039 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:02:08 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:02:08.090849 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44001
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:44001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:08.394506 9369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:08.394760 9369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:08.394891 9369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:08.401608 9369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:08.401710 9369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:08.401788 9369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:08.401866 9369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:08.401924 9369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:08.410893 9369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:44001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:08.411948 9369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:08.413076 9369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:08.423485 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:08.423516 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:08.423540 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:08.424463 9369 server_base.cc:1034] running on GCE node
I20260809 15:02:08.425110 9369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:08.426348 9369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:08.427544 9369 hybrid_clock.cc:648] HybridClock initialized: now 1786287728427496 us; error 37 us; skew 500 ppm
I20260809 15:02:08.427841 9369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:08 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:08.430826 9369 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:02:08.432221 9369 webserver.cc:460] Webserver started at http://127.27.186.190:43707/ using document root <none> and password file <none>
I20260809 15:02:08.432821 9369 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:08.432935 9369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:08.433233 9369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:08.435943 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "948dd7a184cf4fa3843f10a9da597d37"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "4718941d7cad41540ab18f687373219a"
server_key_iv: "29875648be3443bafbbeeb7596f4401e"
server_key_version: "encryptionkey@0"
I20260809 15:02:08.436633 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "948dd7a184cf4fa3843f10a9da597d37"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "4718941d7cad41540ab18f687373219a"
server_key_iv: "29875648be3443bafbbeeb7596f4401e"
server_key_version: "encryptionkey@0"
I20260809 15:02:08.441463 9369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260809 15:02:08.444659 9392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:08.445880 9369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:02:08.446084 9369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "948dd7a184cf4fa3843f10a9da597d37"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "4718941d7cad41540ab18f687373219a"
server_key_iv: "29875648be3443bafbbeeb7596f4401e"
server_key_version: "encryptionkey@0"
I20260809 15:02:08.446269 9369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:08.473994 9369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:08.477406 9369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:08.477638 9369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:08.493857 9369 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:44001
I20260809 15:02:08.493880 9469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:44001 every 8 connection(s)
I20260809 15:02:08.495128 9369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:02:08.497788 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 9369
I20260809 15:02:08.497970 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:02:08.498540 9470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:08.498613 28394 external_mini_cluster.cc:1428] Setting key 6d32be3756876b7e209ba54259590bb0
I20260809 15:02:08.511812 9470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Bootstrap starting.
Aug 09 15:02:08 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:08.515316 9470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:08.516575 9470 log.cc:824] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:08.519646 9470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: No bootstrap required, opened a new log
I20260809 15:02:08.521396 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:08.500348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.27.186.190:44001)
0809 15:02:08.501202 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:08.501221 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:08.501286 (+ 65us) server_negotiation.cc:407] Connection header received
0809 15:02:08.503124 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:08.503150 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:08.503616 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:08.504010 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:08.505217 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:08.507173 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:08.508358 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:08.509381 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:08.509556 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:08.513373 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:08.513388 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:08.513406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:08.513448 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:08.517072 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:08.517889 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:08.517897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:08.517904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:08.518124 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:08.518460 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:08.518464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:08.518467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:08.519087 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:08.519364 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:08.519643 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 15:02:08.520205 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":361,"threads_started":1}
I20260809 15:02:08.528162 9470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:08.528456 9470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:08.528525 9470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948dd7a184cf4fa3843f10a9da597d37, State: Initialized, Role: FOLLOWER
I20260809 15:02:08.529006 9470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:08.529203 9470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:02:08.529306 9470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:02:08.529428 9470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:08.531365 9470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:08.531857 9470 leader_election.cc:302] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 948dd7a184cf4fa3843f10a9da597d37; no voters:
I20260809 15:02:08.532456 9470 leader_election.cc:288] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:02:08.532509 9480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:08.533149 9480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 LEADER]: Becoming Leader. State: Replica: 948dd7a184cf4fa3843f10a9da597d37, State: Running, Role: LEADER
I20260809 15:02:08.533658 9480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:08.534281 9470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:02:08.536204 9481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "948dd7a184cf4fa3843f10a9da597d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } } }
I20260809 15:02:08.536722 9481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:08.537366 9483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948dd7a184cf4fa3843f10a9da597d37. Latest consensus state: current_term: 1 leader_uuid: "948dd7a184cf4fa3843f10a9da597d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } } }
I20260809 15:02:08.537748 9483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:08.538121 9494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:02:08.542886 9494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:02:08.550609 9494 catalog_manager.cc:1347] Generated new cluster ID: f611587c6f034edd8e201bfe1c9bf104
I20260809 15:02:08.550704 9494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:02:08.559057 9494 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:02:08.560689 9494 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:02:08.569373 9494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Generated new TSK 0
I20260809 15:02:08.570214 9494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:02:08.622119 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:44001
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:08.936846 9514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:08.937084 9514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:08.937180 9514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:08.944382 9514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:08.944476 9514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:08.944585 9514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:02:08.953743 9514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:44001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:08.954862 9514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:08.956012 9514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:08.968406 9526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:08.968467 9531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:08.968405 9527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:08.969167 9514 server_base.cc:1034] running on GCE node
I20260809 15:02:08.969717 9514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:08.970599 9514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:08.971778 9514 hybrid_clock.cc:648] HybridClock initialized: now 1786287728971734 us; error 43 us; skew 500 ppm
I20260809 15:02:08.972046 9514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:08 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:08.979552 9514 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:02:08.980988 9514 webserver.cc:460] Webserver started at http://127.27.186.129:38579/ using document root <none> and password file <none>
I20260809 15:02:08.981603 9514 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:08.981717 9514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:08.981999 9514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:08.984719 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "c7b537f66d0a4c5092101270001e46e8"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "c1ecf7113b4f6767e94ab8ae9c6b97d0"
server_key_iv: "e2fb8b2bf9b791bc07553e95b4a31e0b"
server_key_version: "encryptionkey@0"
I20260809 15:02:08.985466 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "c7b537f66d0a4c5092101270001e46e8"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "c1ecf7113b4f6767e94ab8ae9c6b97d0"
server_key_iv: "e2fb8b2bf9b791bc07553e95b4a31e0b"
server_key_version: "encryptionkey@0"
I20260809 15:02:08.990264 9514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260809 15:02:08.993331 9542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:08.994525 9514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:02:08.994683 9514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "c7b537f66d0a4c5092101270001e46e8"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "c1ecf7113b4f6767e94ab8ae9c6b97d0"
server_key_iv: "e2fb8b2bf9b791bc07553e95b4a31e0b"
server_key_version: "encryptionkey@0"
I20260809 15:02:08.994870 9514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:09.010349 9514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:09.014178 9514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:09.014426 9514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:09.015260 9514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:09.016480 9514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:09.016558 9514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.016680 9514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:09.016734 9514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.046152 9514 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:46845
I20260809 15:02:09.046187 9741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:46845 every 8 connection(s)
I20260809 15:02:09.047256 9514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:02:09.049650 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 9514
I20260809 15:02:09.049851 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:02:09.050625 28394 external_mini_cluster.cc:1428] Setting key ebc6dd3b11654d4dc3609284b641bdfa
Aug 09 15:02:09 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:09.067430 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:09.049513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:36897 (local address 127.27.186.190:44001)
0809 15:02:09.049969 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:09.049976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:09.051079 (+ 1103us) server_negotiation.cc:407] Connection header received
0809 15:02:09.052940 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:09.052944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:09.053007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:09.053093 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:09.054904 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.055630 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.057539 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.057921 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.058005 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:09.061639 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:09.061645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:09.061648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:09.061680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:09.064535 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:09.065199 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:09.065205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:09.065209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:09.065346 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:09.065713 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:09.065718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:09.065720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:09.066093 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:09.066237 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:09.066830 (+ 593us) server_negotiation.cc:299] Negotiation successful
0809 15:02:09.067066 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":242,"threads_started":1}
I20260809 15:02:09.067919 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:09.050044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:36897)
0809 15:02:09.050920 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:09.050967 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:09.052649 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:09.053349 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:09.053363 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:09.054004 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:09.054732 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:09.054754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.055790 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.055795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:09.057423 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:09.057437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.058147 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.058154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:09.058403 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:09.058987 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:09.059012 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:09.061445 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:09.064661 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:09.064667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:09.064676 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:09.065066 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:09.065440 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:09.065443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:09.065445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:09.065602 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:09.066213 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:09.066218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:09.066483 (+ 265us) client_negotiation.cc:769] Sending connection context
0809 15:02:09.066686 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 15:02:09.067011 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":316,"threads_started":1}
I20260809 15:02:09.069345 9742 heartbeater.cc:347] Connected to a master server at 127.27.186.190:44001
I20260809 15:02:09.069619 9742 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:09.070319 9742 heartbeater.cc:511] Master 127.27.186.190:44001 requested a full tablet report, sending...
I20260809 15:02:09.072395 9416 ts_manager.cc:194] Registered new tserver with Master: c7b537f66d0a4c5092101270001e46e8 (127.27.186.129:46845)
I20260809 15:02:09.074374 9416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:36897
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:02:09.100903 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:44001
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:09.408835 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:09.409068 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:09.409192 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:09.416015 9749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:09.416108 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:09.416262 9749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:02:09.425328 9749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:44001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:09.426373 9749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:09.427574 9749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:09.439687 9758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:09.439646 9762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:09.439721 9759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:09.440323 9749 server_base.cc:1034] running on GCE node
I20260809 15:02:09.440872 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:09.441668 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:09.442868 9749 hybrid_clock.cc:648] HybridClock initialized: now 1786287729442804 us; error 53 us; skew 500 ppm
I20260809 15:02:09.443149 9749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:09 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287729, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:09.445957 9749 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:02:09.447290 9749 webserver.cc:460] Webserver started at http://127.27.186.130:43391/ using document root <none> and password file <none>
I20260809 15:02:09.447906 9749 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:09.448020 9749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:09.448302 9749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:09.451027 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "bd2e00e5ab8b4f3db577ea6ccc0e5dba"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "02baae117cf6e6a7208632420e8e7ab4"
server_key_iv: "3258eae9b51b20d25a88496a2e2ac94c"
server_key_version: "encryptionkey@0"
I20260809 15:02:09.451761 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "bd2e00e5ab8b4f3db577ea6ccc0e5dba"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "02baae117cf6e6a7208632420e8e7ab4"
server_key_iv: "3258eae9b51b20d25a88496a2e2ac94c"
server_key_version: "encryptionkey@0"
I20260809 15:02:09.456540 9749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 15:02:09.460163 9773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.461592 9749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:02:09.461745 9749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "bd2e00e5ab8b4f3db577ea6ccc0e5dba"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "02baae117cf6e6a7208632420e8e7ab4"
server_key_iv: "3258eae9b51b20d25a88496a2e2ac94c"
server_key_version: "encryptionkey@0"
I20260809 15:02:09.461918 9749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:09.489979 9749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:09.493486 9749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:09.493750 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:09.494652 9749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:09.495931 9749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:09.496009 9749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.496125 9749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:09.496179 9749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.528707 9749 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33099
I20260809 15:02:09.528755 9967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33099 every 8 connection(s)
I20260809 15:02:09.529868 9749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:02:09.537426 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 9749
I20260809 15:02:09.537606 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:02:09.538175 28394 external_mini_cluster.cc:1428] Setting key 2890843b56dccc8d0aac186824a4509e
Aug 09 15:02:09 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287729, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:09.549674 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:09.532095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:60643 (local address 127.27.186.190:44001)
0809 15:02:09.532245 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:09.532250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:09.533343 (+ 1093us) server_negotiation.cc:407] Connection header received
0809 15:02:09.535166 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:09.535171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:09.535257 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:09.535342 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:09.536679 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.537383 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.539378 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.539986 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.540114 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:09.543958 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:09.543964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:09.543969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:09.544005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:09.546716 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:09.547494 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:09.547498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:09.547501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:09.547634 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:09.548056 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:09.548059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:09.548062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:09.548338 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:09.548452 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:09.549213 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 15:02:09.549392 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:02:09.550783 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:09.532529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:60643)
0809 15:02:09.533181 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:09.533223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:09.534924 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:09.535546 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:09.535555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:09.536035 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:09.536553 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:09.536569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.537533 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.537539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:09.539243 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:09.539256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:09.540196 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:09.540203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:09.540483 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:09.541134 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:09.541161 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:09.543733 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:09.546852 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:09.546860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:09.546873 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:09.547394 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:09.547727 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:09.547731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:09.547734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:09.547957 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:09.548451 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:09.548457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:09.548807 (+ 350us) client_negotiation.cc:769] Sending connection context
0809 15:02:09.549102 (+ 295us) client_negotiation.cc:240] Negotiation successful
0809 15:02:09.549596 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260809 15:02:09.552196 9968 heartbeater.cc:347] Connected to a master server at 127.27.186.190:44001
I20260809 15:02:09.552466 9968 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:09.553088 9968 heartbeater.cc:511] Master 127.27.186.190:44001 requested a full tablet report, sending...
I20260809 15:02:09.554374 9416 ts_manager.cc:194] Registered new tserver with Master: bd2e00e5ab8b4f3db577ea6ccc0e5dba (127.27.186.130:33099)
I20260809 15:02:09.555130 9416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:60643
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:02:09.588867 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:44001
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:09.902878 9974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:09.903129 9974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:09.903259 9974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:09.909988 9974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:09.910108 9974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:09.910261 9974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:02:09.919158 9974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:44001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:09.920207 9974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:09.921389 9974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:09.933346 9988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:09.933481 9983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:09.933478 9985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:09.934346 9974 server_base.cc:1034] running on GCE node
I20260809 15:02:09.934948 9974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:09.935806 9974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:09.937000 9974 hybrid_clock.cc:648] HybridClock initialized: now 1786287729936950 us; error 46 us; skew 500 ppm
I20260809 15:02:09.937285 9974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:09 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287729, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:09.940313 9974 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:02:09.941577 9974 webserver.cc:460] Webserver started at http://127.27.186.131:35965/ using document root <none> and password file <none>
I20260809 15:02:09.942225 9974 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:09.942348 9974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:09.942624 9974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:09.945298 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "0425a940b8a741e78121be441eec889f"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "2aebf309fdc6996968fb6c774de9cb1a"
server_key_iv: "29d23cf8f6730ac4b12bbd92fe29d56b"
server_key_version: "encryptionkey@0"
I20260809 15:02:09.946062 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "0425a940b8a741e78121be441eec889f"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "2aebf309fdc6996968fb6c774de9cb1a"
server_key_iv: "29d23cf8f6730ac4b12bbd92fe29d56b"
server_key_version: "encryptionkey@0"
I20260809 15:02:09.950838 9974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:02:09.953956 9998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.955168 9974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:02:09.955300 9974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "0425a940b8a741e78121be441eec889f"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "2aebf309fdc6996968fb6c774de9cb1a"
server_key_iv: "29d23cf8f6730ac4b12bbd92fe29d56b"
server_key_version: "encryptionkey@0"
I20260809 15:02:09.955485 9974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:09.967458 9974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:09.971333 9974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:09.971598 9974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:09.972440 9974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:09.973733 9974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:09.973812 9974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:09.973909 9974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:09.973958 9974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:10.002312 9974 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:44393
I20260809 15:02:10.002341 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:44393 every 8 connection(s)
I20260809 15:02:10.003568 9974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:02:10.008029 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 9974
I20260809 15:02:10.008178 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:02:10.008574 28394 external_mini_cluster.cc:1428] Setting key 00c1d923d7ecb34342d1465d67c3e130
Aug 09 15:02:10 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287729, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:10.022380 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:10.005965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:46001 (local address 127.27.186.190:44001)
0809 15:02:10.006171 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:10.006176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:10.007296 (+ 1120us) server_negotiation.cc:407] Connection header received
0809 15:02:10.009302 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:10.009308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:10.009391 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:10.009501 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:10.011384 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.012196 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.013676 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.014115 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.014206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:10.017205 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:10.017210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:10.017213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:10.017244 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:10.019673 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.020338 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.020341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.020344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.020454 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.020785 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.020788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.020790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.021038 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:10.021142 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:10.021841 (+ 699us) server_negotiation.cc:299] Negotiation successful
0809 15:02:10.022079 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 15:02:10.023291 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:10.006410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:46001)
0809 15:02:10.007132 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:10.007182 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:10.008975 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:10.009787 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:10.009798 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:10.010455 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:10.011185 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:10.011212 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.012335 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.012339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:10.013555 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:10.013568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.014282 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.014286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:10.014463 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:10.015016 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:10.015037 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:10.017021 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:10.019807 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:10.019815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:10.019825 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:10.020243 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:10.020527 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:10.020530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:10.020532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:10.020700 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:10.021172 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:10.021178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:10.021449 (+ 271us) client_negotiation.cc:769] Sending connection context
0809 15:02:10.021751 (+ 302us) client_negotiation.cc:240] Negotiation successful
0809 15:02:10.022191 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":273,"threads_started":1}
I20260809 15:02:10.024501 10185 heartbeater.cc:347] Connected to a master server at 127.27.186.190:44001
I20260809 15:02:10.024793 10185 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:10.025441 10185 heartbeater.cc:511] Master 127.27.186.190:44001 requested a full tablet report, sending...
I20260809 15:02:10.026759 9416 ts_manager.cc:194] Registered new tserver with Master: 0425a940b8a741e78121be441eec889f (127.27.186.131:44393)
I20260809 15:02:10.027529 9416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:46001
I20260809 15:02:10.035696 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.129@KRBTEST.COM" created.
I20260809 15:02:10.077088 9742 heartbeater.cc:503] Master 127.27.186.190:44001 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab.
Entry for principal oryx/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab.
WARNING: no policy specified for oryx/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab.
Entry for principal oryx/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab.
WARNING: no policy specified for oryx/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab.
Entry for principal oryx/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab.
I20260809 15:02:10.167670 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 9514
Aug 09 15:02:10 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:02:10.182853 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:10.168662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.27.186.129:46845)
0809 15:02:10.169394 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:10.169401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:10.169429 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:02:10.169490 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:10.169497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:10.169705 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:10.169842 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:10.171080 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.172085 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.173244 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.173784 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.173879 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:10.177569 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:10.177581 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:10.177595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:10.177632 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:10.180516 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.181102 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.181108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.181111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.181246 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.181512 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.181515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.181517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.181836 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:10.182105 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:10.182281 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 15:02:10.182486 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":408,"threads_started":1}
I20260809 15:02:10.184008 9731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:10.265462 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 9514
I20260809 15:02:10.286523 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 9749
Aug 09 15:02:10 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:02:10.301343 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:10.287447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.27.186.130:33099)
0809 15:02:10.287938 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:10.287945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:10.287972 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:02:10.288034 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:10.288040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:10.288252 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:10.288381 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:10.289532 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.290535 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.291762 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.292307 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.292402 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:10.296380 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:10.296391 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:10.296404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:10.296441 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:10.299030 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.299608 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.299614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.299616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.299740 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.299993 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.299996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.299998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.300324 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:10.300546 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:10.300762 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 15:02:10.300950 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
I20260809 15:02:10.302556 9960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:10.382721 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 9749
I20260809 15:02:10.404196 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 9974
Aug 09 15:02:10 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:02:10.419059 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:10.405042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.27.186.131:44393)
0809 15:02:10.405650 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:10.405657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:10.405686 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:02:10.405749 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:10.405755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:10.405961 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:10.406119 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:10.407295 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.408285 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.409523 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:10.410108 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:10.410224 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:10.414276 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:10.414288 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:10.414302 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:10.414338 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:10.416828 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.417409 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.417415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.417417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.417540 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:10.417819 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:10.417821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:10.417823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:10.418175 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:10.418422 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:10.418601 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 15:02:10.418790 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":297,"threads_started":1}
I20260809 15:02:10.420271 10179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:10.501281 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 9974
I20260809 15:02:10.523406 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 9369
I20260809 15:02:10.524434 9461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:10.623019 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 9369
I20260809 15:02:10.650066 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44001
--webserver_interface=127.27.186.190
--webserver_port=43707
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:44001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:10.955770 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:10.956017 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:10.956118 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:10.962917 10206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:10.963006 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:10.963057 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:10.963117 10206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:10.963160 10206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:10.972182 10206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:44001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=43707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:10.973182 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:10.974290 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:10.984956 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:10.985042 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:10.984987 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:10.985867 10206 server_base.cc:1034] running on GCE node
I20260809 15:02:10.986603 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:10.987744 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:10.988940 10206 hybrid_clock.cc:648] HybridClock initialized: now 1786287730988902 us; error 35 us; skew 500 ppm
I20260809 15:02:10.989249 10206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:10 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287730, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:10.992053 10206 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:02:10.993372 10206 webserver.cc:460] Webserver started at http://127.27.186.190:43707/ using document root <none> and password file <none>
I20260809 15:02:10.993959 10206 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:10.994118 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:10.998953 10206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:02:11.001422 10231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.002592 10206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:02:11.002740 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "948dd7a184cf4fa3843f10a9da597d37"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "4718941d7cad41540ab18f687373219a"
server_key_iv: "29875648be3443bafbbeeb7596f4401e"
server_key_version: "encryptionkey@0"
I20260809 15:02:11.003592 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:11.025509 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:11.029192 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:11.029412 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:11.045665 10206 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:44001
I20260809 15:02:11.045713 10308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:44001 every 8 connection(s)
I20260809 15:02:11.046921 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:02:11.049960 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 10206
I20260809 15:02:11.050747 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:46845
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:44001
--webserver_port=38579
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:02:11.053176 10310 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:02:11.056197 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Bootstrap starting.
I20260809 15:02:11.074671 10310 log.cc:824] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:11.082794 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:02:11.083251 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Bootstrap complete.
I20260809 15:02:11.091732 10310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:11.092505 10310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948dd7a184cf4fa3843f10a9da597d37, State: Initialized, Role: FOLLOWER
I20260809 15:02:11.093022 10310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:11.093252 10310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:02:11.093354 10310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:02:11.093508 10310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:02:11.096974 10310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:11.097420 10310 leader_election.cc:302] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 948dd7a184cf4fa3843f10a9da597d37; no voters:
I20260809 15:02:11.097944 10310 leader_election.cc:288] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 15:02:11.098150 10317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:02:11.099176 10317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [term 2 LEADER]: Becoming Leader. State: Replica: 948dd7a184cf4fa3843f10a9da597d37, State: Running, Role: LEADER
I20260809 15:02:11.099900 10310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:02:11.099826 10317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } }
I20260809 15:02:11.103597 10318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "948dd7a184cf4fa3843f10a9da597d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } } }
I20260809 15:02:11.103987 10318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:11.104472 10319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948dd7a184cf4fa3843f10a9da597d37. Latest consensus state: current_term: 2 leader_uuid: "948dd7a184cf4fa3843f10a9da597d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948dd7a184cf4fa3843f10a9da597d37" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44001 } } }
I20260809 15:02:11.104786 10319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37 [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:11.115221 10335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:02:11.120486 10335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:02:11.121933 10335 catalog_manager.cc:1259] Loaded cluster ID: f611587c6f034edd8e201bfe1c9bf104
I20260809 15:02:11.122128 10335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:02:11.126226 10335 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:02:11.127063 10349 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 15:02:11.127774 10335 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 948dd7a184cf4fa3843f10a9da597d37: Loaded TSK: 0
I20260809 15:02:11.128880 10335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 15:02:11.394876 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:11.395109 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:11.395223 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:11.402218 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:11.402315 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:11.402465 10315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:02:11.411628 10315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.129.keytab
--principal=oryx/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:46845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=38579
--tserver_master_addrs=127.27.186.190:44001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:11.412744 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:11.413923 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:11.427870 10357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:11.427812 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:11.428395 10362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:11.428905 10315 server_base.cc:1034] running on GCE node
I20260809 15:02:11.429436 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:11.430341 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:11.431555 10315 hybrid_clock.cc:648] HybridClock initialized: now 1786287731431503 us; error 48 us; skew 500 ppm
I20260809 15:02:11.431835 10315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287731, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:11.434813 10315 init.cc:377] Logged in from keytab as oryx/127.27.186.129@KRBTEST.COM (short username oryx)
I20260809 15:02:11.436228 10315 webserver.cc:460] Webserver started at http://127.27.186.129:38579/ using document root <none> and password file <none>
I20260809 15:02:11.436836 10315 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:11.436960 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:11.442009 10315 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:02:11.444679 10372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.446075 10315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:02:11.446221 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "c7b537f66d0a4c5092101270001e46e8"
format_stamp: "Formatted at 2026-08-09 15:02:08 on dist-test-slave-q9h9"
server_key: "c1ecf7113b4f6767e94ab8ae9c6b97d0"
server_key_iv: "e2fb8b2bf9b791bc07553e95b4a31e0b"
server_key_version: "encryptionkey@0"
I20260809 15:02:11.447089 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:11.473094 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:11.476603 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:11.476842 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:11.477734 10315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:11.479022 10315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:11.479127 10315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.479228 10315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:11.479306 10315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.510047 10315 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:46845
I20260809 15:02:11.510142 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:46845 every 8 connection(s)
I20260809 15:02:11.511346 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:02:11.514006 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 10315
I20260809 15:02:11.514864 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:33099
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:44001
--webserver_port=43391
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:11.529618 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:11.513912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:34057)
0809 15:02:11.514710 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:11.514753 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:11.516562 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:11.519509 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:11.519520 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:11.520084 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:11.520569 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.520586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.522158 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.522162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.523311 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.523324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.524293 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.524301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.524550 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.525145 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:11.525168 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:11.528485 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
W20260809 15:02:11.529788 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:11.513952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:34057 (local address 127.27.186.190:44001)
0809 15:02:11.518419 (+ 4467us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:11.518430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:11.518466 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:02:11.518733 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:11.518745 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:11.519030 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:11.519338 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:11.520764 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.521982 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.523451 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.524073 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.524257 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.528819 (+ 4562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:34057: BlockingRecv error: recv got EOF from 127.27.186.129:34057 (error 108)
Metrics: {"server-negotiator.queue_time_us":4270,"thread_start_us":4064,"threads_started":1}
W20260809 15:02:11.530517 10567 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:44001 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.186.190:44001: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:11.537850 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:11.531037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:44019)
0809 15:02:11.531207 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:11.531221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:11.531294 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:11.531620 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:11.531624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:11.531926 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:11.532055 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.532062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.533014 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.533019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.533861 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.533872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.534542 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.534548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.534706 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.535246 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:11.535261 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:11.537604 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 15:02:11.538125 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:11.531248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:44019 (local address 127.27.186.190:44001)
0809 15:02:11.531408 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:11.531412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:11.531424 (+ 12us) server_negotiation.cc:407] Connection header received
0809 15:02:11.531459 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:11.531462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:11.531507 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:11.531580 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:11.532180 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.532885 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.533986 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.534414 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.534518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.537806 (+ 3288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:44019: BlockingRecv error: recv got EOF from 127.27.186.129:44019 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:11.545507 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:11.538667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:42099)
0809 15:02:11.538812 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:11.538826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:11.538897 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:11.539245 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:11.539249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:11.539542 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:11.539682 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.539690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.540628 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.540633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.541502 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.541513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.542196 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.542202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.542348 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.542828 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:11.542842 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:11.545131 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 15:02:11.545537 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:11.538849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:42099 (local address 127.27.186.190:44001)
0809 15:02:11.539002 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:11.539008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:11.539022 (+ 14us) server_negotiation.cc:407] Connection header received
0809 15:02:11.539056 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:11.539058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:11.539117 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:11.539191 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:11.539806 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.540497 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.541625 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.542055 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.542153 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.545324 (+ 3171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:42099: BlockingRecv error: recv got EOF from 127.27.186.129:42099 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 15:02:11.545850 10567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 15:02:11.840023 10572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:11.840296 10572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:11.840438 10572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:11.847297 10572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:11.847402 10572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:11.847549 10572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:02:11.856760 10572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.130.keytab
--principal=oryx/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:33099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=43391
--tserver_master_addrs=127.27.186.190:44001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:11.857859 10572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:11.859109 10572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:11.870965 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:11.871008 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:11.871126 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:11.871942 10572 server_base.cc:1034] running on GCE node
I20260809 15:02:11.872476 10572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:11.873299 10572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:11.874490 10572 hybrid_clock.cc:648] HybridClock initialized: now 1786287731874448 us; error 35 us; skew 500 ppm
I20260809 15:02:11.874773 10572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287731, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:11.877628 10572 init.cc:377] Logged in from keytab as oryx/127.27.186.130@KRBTEST.COM (short username oryx)
I20260809 15:02:11.878978 10572 webserver.cc:460] Webserver started at http://127.27.186.130:43391/ using document root <none> and password file <none>
I20260809 15:02:11.879572 10572 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:11.879706 10572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:11.884624 10572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 15:02:11.887151 10598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.888362 10572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:02:11.888504 10572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "bd2e00e5ab8b4f3db577ea6ccc0e5dba"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "02baae117cf6e6a7208632420e8e7ab4"
server_key_iv: "3258eae9b51b20d25a88496a2e2ac94c"
server_key_version: "encryptionkey@0"
I20260809 15:02:11.889364 10572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:11.908972 10572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:11.912587 10572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:11.912827 10572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:11.913690 10572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:11.914980 10572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:11.915077 10572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.915184 10572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:11.915256 10572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:11.944958 10572 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33099
I20260809 15:02:11.945032 10784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33099 every 8 connection(s)
I20260809 15:02:11.946323 10572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:02:11.950011 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 10572
I20260809 15:02:11.950757 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:44393
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:44001
--webserver_port=35965
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
W20260809 15:02:11.963102 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:11.948616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:59845 (local address 127.27.186.190:44001)
0809 15:02:11.948796 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:11.948801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:11.949904 (+ 1103us) server_negotiation.cc:407] Connection header received
0809 15:02:11.952312 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:11.952317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:11.952384 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:11.952468 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:11.954274 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.955232 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.957276 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.957820 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.957928 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.962861 (+ 4933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:59845: BlockingRecv error: recv got EOF from 127.27.186.130:59845 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:02:11.963660 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:11.948998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:59845)
0809 15:02:11.949730 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:11.949790 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:11.952013 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:11.952721 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:11.952731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:11.953374 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:11.954076 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.954099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.955442 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.955447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.957115 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.957143 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.958074 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.958082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.958347 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.958963 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:11.958995 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:11.962670 (+ 3675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":267,"threads_started":1}
W20260809 15:02:11.964269 10785 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:44001 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.186.190:44001: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:11.972506 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:11.964754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:54029)
0809 15:02:11.964922 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:11.964942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:11.965044 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:11.965316 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:11.965320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:11.965684 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:11.965843 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.965853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.966839 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.966844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.968107 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.968122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.968834 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.968839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.968992 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.969504 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:11.969523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:11.972331 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 15:02:11.973095 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:11.964881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:54029 (local address 127.27.186.190:44001)
0809 15:02:11.965016 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:11.965020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:11.965035 (+ 15us) server_negotiation.cc:407] Connection header received
0809 15:02:11.965149 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:11.965152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:11.965202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:11.965272 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:11.965987 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.966703 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.968279 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.968684 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.968784 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.972581 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:54029: BlockingRecv error: recv got EOF from 127.27.186.130:54029 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:11 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:11.981742 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:11.973059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:51761)
0809 15:02:11.973257 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:11.973274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:11.973362 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:11.973660 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:11.973665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:11.974071 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:11.974256 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.974267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.975608 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.975613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.977092 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:11.977105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.978091 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.978097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:11.978276 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.978856 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:11.978872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:11.981466 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260809 15:02:11.981773 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:11.973164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:51761 (local address 127.27.186.190:44001)
0809 15:02:11.973378 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:11.973384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:11.973404 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:02:11.973453 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:11.973457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:11.973525 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:11.973627 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:11.974411 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.975479 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.977245 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:11.977917 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:11.978083 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:11.981616 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:51761: BlockingRecv error: recv got EOF from 127.27.186.130:51761 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":5425,"threads_started":1}
W20260809 15:02:11.982107 10785 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 15:02:12.276479 10788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:12.276712 10788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:12.276813 10788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:12.283725 10788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:12.283818 10788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:12.283929 10788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:02:12.293247 10788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/krb5kdc/oryx_127.27.186.131.keytab
--principal=oryx/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:44393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=35965
--tserver_master_addrs=127.27.186.190:44001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:12.294332 10788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:12.295465 10788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:12.307328 10799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:12.307219 10801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:12.307442 10804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:12.308252 10788 server_base.cc:1034] running on GCE node
I20260809 15:02:12.308806 10788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:12.309707 10788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:12.310891 10788 hybrid_clock.cc:648] HybridClock initialized: now 1786287732310842 us; error 42 us; skew 500 ppm
I20260809 15:02:12.311163 10788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287732, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:12.313999 10788 init.cc:377] Logged in from keytab as oryx/127.27.186.131@KRBTEST.COM (short username oryx)
I20260809 15:02:12.315311 10788 webserver.cc:460] Webserver started at http://127.27.186.131:35965/ using document root <none> and password file <none>
I20260809 15:02:12.315912 10788 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:12.316035 10788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:12.320888 10788 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260809 15:02:12.323318 10820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:12.324502 10788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:02:12.324661 10788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "0425a940b8a741e78121be441eec889f"
format_stamp: "Formatted at 2026-08-09 15:02:09 on dist-test-slave-q9h9"
server_key: "2aebf309fdc6996968fb6c774de9cb1a"
server_key_iv: "29d23cf8f6730ac4b12bbd92fe29d56b"
server_key_version: "encryptionkey@0"
I20260809 15:02:12.325543 10788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:12.355849 10788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:12.359447 10788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:12.359674 10788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:12.360548 10788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:12.361742 10788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:12.361853 10788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:12.361958 10788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:12.362048 10788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:12.391374 10788 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:44393
I20260809 15:02:12.391433 10987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:44393 every 8 connection(s)
I20260809 15:02:12.392699 10788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:02:12.396885 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 10788
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
W20260809 15:02:12.409972 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:12.395150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:58427 (local address 127.27.186.190:44001)
0809 15:02:12.395316 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:12.395321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:12.396414 (+ 1093us) server_negotiation.cc:407] Connection header received
0809 15:02:12.398795 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:12.398801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:12.398900 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:12.399006 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:12.400976 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.402056 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.404191 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.404658 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.404760 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.409675 (+ 4915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:58427: BlockingRecv error: recv got EOF from 127.27.186.131:58427 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":16512}
I20260809 15:02:12.410513 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:12.395593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:58427)
0809 15:02:12.396270 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:12.396308 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:12.398520 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:12.399251 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:12.399262 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:12.399964 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:12.400762 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.400786 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.402210 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.402215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.404054 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.404065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.404864 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.404872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.405077 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.405600 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:12.405620 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:12.409481 (+ 3861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":267,"threads_started":1}
W20260809 15:02:12.411033 10988 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:44001 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.186.190:44001: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
I20260809 15:02:12.413990 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:12.397896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.27.186.190:44001)
0809 15:02:12.398074 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:12.398080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:12.398622 (+ 542us) server_negotiation.cc:407] Connection header received
0809 15:02:12.398795 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:12.398801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:12.398900 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:12.399006 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:12.400955 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.402106 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.403898 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.404325 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.404452 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.407223 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:12.407234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:12.407251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:12.407293 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:12.410661 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:12.411421 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:12.411427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:12.411433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:12.411654 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:12.412116 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:12.412122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:12.412125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:12.412749 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:12.413027 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:12.413379 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 15:02:12.413630 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":28800}
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:12.419704 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:12.411527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:54037)
0809 15:02:12.411689 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:12.411706 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:12.411786 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:12.412131 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:12.412136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:12.412530 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:12.412715 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.412728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.413978 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.413983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.415320 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.415332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.416259 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.416265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.416420 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.416868 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:12.416887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:12.419462 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 15:02:12.419852 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:12.411629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:54037 (local address 127.27.186.190:44001)
0809 15:02:12.411766 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:12.411779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:12.411795 (+ 16us) server_negotiation.cc:407] Connection header received
0809 15:02:12.411891 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:12.411896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:12.411960 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:12.412055 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:12.412850 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.413847 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.415449 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.416125 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.416229 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.419645 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:54037: BlockingRecv error: recv got EOF from 127.27.186.131:54037 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:12.426703 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:12.420284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:54753)
0809 15:02:12.420409 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:12.420422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:12.420499 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:12.420827 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:12.420831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:12.421090 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:12.421240 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.421249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.422197 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.422201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.423078 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.423089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.423736 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.423741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.423889 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.424281 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:12.424295 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:12.426516 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 15:02:12.426862 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:12.420381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:54753 (local address 127.27.186.190:44001)
0809 15:02:12.420617 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:12.420621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:12.420634 (+ 13us) server_negotiation.cc:407] Connection header received
0809 15:02:12.420666 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:12.420669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:12.420715 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:12.420788 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:12.421368 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.422076 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.423216 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.423600 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.423701 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.426664 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:54753: BlockingRecv error: recv got EOF from 127.27.186.131:54753 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260809 15:02:12.426951 10988 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:12.554170 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:12.546375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:32863)
0809 15:02:12.546837 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:12.546853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:12.546961 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:12.547303 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:12.547308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:12.547758 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:12.547968 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.547982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.548953 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.548957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.549925 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.549938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.550697 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.550703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.550855 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.551341 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:12.551358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:12.553907 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
W20260809 15:02:12.554368 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:12.546444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:32863 (local address 127.27.186.190:44001)
0809 15:02:12.546588 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:12.546592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:12.546914 (+ 322us) server_negotiation.cc:407] Connection header received
0809 15:02:12.547104 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:12.547110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:12.547171 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:12.547248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:12.548114 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.548813 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.550075 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.550515 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.550623 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.554164 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:32863: BlockingRecv error: recv got EOF from 127.27.186.129:32863 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:12 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:12.990576 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:12.982584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:40597)
0809 15:02:12.983039 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:12.983054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:12.983172 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:12.983487 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:12.983492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:12.983940 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:12.984123 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.984132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.985098 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.985104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.986083 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:12.986096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.986803 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.986810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:12.986968 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.987494 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:12.987511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:12.990307 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
W20260809 15:02:12.990761 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:12.982652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:40597 (local address 127.27.186.190:44001)
0809 15:02:12.982868 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:12.982874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:12.983133 (+ 259us) server_negotiation.cc:407] Connection header received
0809 15:02:12.983292 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:12.983298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:12.983360 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:12.983437 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:12.984265 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.984954 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.986217 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:12.986633 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:12.986741 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:12.990519 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:40597: BlockingRecv error: recv got EOF from 127.27.186.130:40597 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 15:02:13 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:13 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:13.435441 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:13.427483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:44169)
0809 15:02:13.427952 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:13.427971 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:13.428096 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:13.428435 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:13.428440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:13.428894 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:13.429079 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:13.429089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.430313 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.430318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:13.431257 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:13.431270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.431957 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.431964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:13.432122 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:13.432603 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:13.432619 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:13.435149 (+ 2530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
W20260809 15:02:13.435590 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:13.427557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:44169 (local address 127.27.186.190:44001)
0809 15:02:13.427755 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:13.427761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:13.428055 (+ 294us) server_negotiation.cc:407] Connection header received
0809 15:02:13.428216 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:13.428222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:13.428288 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:13.428366 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:13.429237 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.430158 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.431395 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.431789 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.431896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:13.435369 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:44169: BlockingRecv error: recv got EOF from 127.27.186.131:44169 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 15:02:13 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:13 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:13.562795 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:13.555026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:53657)
0809 15:02:13.555459 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:13.555474 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:13.555578 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:13.555829 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:13.555831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:13.556301 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:13.556498 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:13.556508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.557803 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.557807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:13.558703 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:13.558725 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.559354 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.559358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:13.559467 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:13.559971 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:13.559989 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:13.562556 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
W20260809 15:02:13.562968 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:13.555104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:53657 (local address 127.27.186.190:44001)
0809 15:02:13.555321 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:13.555327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:13.555545 (+ 218us) server_negotiation.cc:407] Connection header received
0809 15:02:13.555663 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:13.555666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:13.555725 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:13.555802 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:13.556640 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.557681 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.558820 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.559228 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.559330 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:13.562729 (+ 3399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:53657: BlockingRecv error: recv got EOF from 127.27.186.129:53657 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 15:02:13 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:13 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:14.003929 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:13.991395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:34987)
0809 15:02:13.991884 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:13.991904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:13.992038 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:13.992369 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:13.992374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:13.992912 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:13.993126 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:13.993140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.994419 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.994423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:13.995324 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:13.995336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.995977 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.995981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:13.996089 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:14.001026 (+ 4937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:14.001043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:14.003695 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":263,"threads_started":1}
W20260809 15:02:14.004072 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:13.991480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:34987 (local address 127.27.186.190:44001)
0809 15:02:13.991656 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:13.991662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:13.991986 (+ 324us) server_negotiation.cc:407] Connection header received
0809 15:02:13.992165 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:13.992168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:13.992229 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:13.992307 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:13.993283 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.994290 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.995452 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:13.995866 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:13.995958 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:14.003888 (+ 7930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:34987: BlockingRecv error: recv got EOF from 127.27.186.130:34987 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 15:02:14 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:14 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:14.444340 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:14.436304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:51569)
0809 15:02:14.436819 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:14.436840 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:14.436961 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:14.437233 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:14.437235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:14.437633 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:14.437814 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:14.437823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.438794 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.438798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:14.439668 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:14.439680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.440327 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.440331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:14.440442 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:14.440973 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:14.440989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:14.444022 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
W20260809 15:02:14.444474 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:14.436479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:51569 (local address 127.27.186.190:44001)
0809 15:02:14.436677 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:14.436681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:14.436913 (+ 232us) server_negotiation.cc:407] Connection header received
0809 15:02:14.437055 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:14.437059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:14.437119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:14.437200 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:14.437925 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.438670 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.439789 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.440204 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.440291 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:14.444254 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:51569: BlockingRecv error: recv got EOF from 127.27.186.131:51569 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 15:02:14 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:14 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:14.571548 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:14.563596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:56783)
0809 15:02:14.564029 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:14.564044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:14.564168 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:14.564485 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:14.564489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:14.564847 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:14.565013 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:14.565024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.566102 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.566107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:14.566991 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:14.567003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.567683 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.567689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:14.567848 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:14.568361 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:14.568378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:14.571216 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
W20260809 15:02:14.571640 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:14.563657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:56783 (local address 127.27.186.190:44001)
0809 15:02:14.563848 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:14.563854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:14.564123 (+ 269us) server_negotiation.cc:407] Connection header received
0809 15:02:14.564279 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:14.564285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:14.564354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:14.564430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:14.565163 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.565928 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.567129 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:14.567541 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:14.567647 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:14.571415 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:56783: BlockingRecv error: recv got EOF from 127.27.186.129:56783 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 15:02:15 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:15 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:15.012567 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:15.004852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:42737)
0809 15:02:15.005259 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:15.005274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:15.005384 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:15.005623 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:15.005626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:15.005959 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:15.006142 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:15.006152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.007199 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.007203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:15.008192 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:15.008206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.008831 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.008835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:15.008956 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:15.009501 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:15.009519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:15.012207 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
W20260809 15:02:15.012641 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:15.004917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:42737 (local address 127.27.186.190:44001)
0809 15:02:15.005113 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:15.005119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:15.005341 (+ 222us) server_negotiation.cc:407] Connection header received
0809 15:02:15.005471 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:15.005474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:15.005535 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:15.005608 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:15.006285 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.007080 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.008290 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.008706 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.008803 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:15.012420 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:42737: BlockingRecv error: recv got EOF from 127.27.186.130:42737 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 15:02:15 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:15 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:15.452847 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:15.445263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:52139)
0809 15:02:15.445714 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:15.445738 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:15.445862 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:15.446144 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:15.446148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:15.446483 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:15.446645 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:15.446654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.447532 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.447535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:15.448421 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:15.448432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.449109 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.449113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:15.449236 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:15.449847 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:15.449863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:15.452528 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":225,"threads_started":1}
W20260809 15:02:15.452919 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:15.445342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:52139 (local address 127.27.186.190:44001)
0809 15:02:15.445509 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:15.445513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:15.445816 (+ 303us) server_negotiation.cc:407] Connection header received
0809 15:02:15.445948 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:15.445951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:15.446046 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:15.446120 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:15.446754 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.447437 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.448558 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.448996 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.449093 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:15.452749 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:52139: BlockingRecv error: recv got EOF from 127.27.186.131:52139 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 15:02:15 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:15 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:15.580020 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:15.572462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:55257)
0809 15:02:15.572956 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:15.572971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:15.573082 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:15.573336 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:15.573339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:15.573694 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:15.573849 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:15.573858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.574751 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.574755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:15.575723 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:15.575736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.576349 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.576353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:15.576469 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:15.576988 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:15.577005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:15.579712 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
W20260809 15:02:15.580168 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:15.572546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:55257 (local address 127.27.186.190:44001)
0809 15:02:15.572704 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:15.572708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:15.573032 (+ 324us) server_negotiation.cc:407] Connection header received
0809 15:02:15.573160 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:15.573163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:15.573223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:15.573308 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:15.573953 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.574662 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.575826 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:15.576249 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:15.576334 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:15.579940 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:55257: BlockingRecv error: recv got EOF from 127.27.186.129:55257 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 15:02:16 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:16 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:16.021065 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:16.013494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:47051)
0809 15:02:16.013897 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:16.013912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:16.014049 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:16.014325 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:16.014328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:16.014687 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:16.014854 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:16.014864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.015807 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.015811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:16.016762 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:16.016774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.017372 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.017376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:16.017490 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:16.018003 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:16.018049 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:16.020848 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":209,"threads_started":1}
W20260809 15:02:16.021207 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:16.013536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:47051 (local address 127.27.186.190:44001)
0809 15:02:16.013742 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:16.013748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:16.013978 (+ 230us) server_negotiation.cc:407] Connection header received
0809 15:02:16.014144 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:16.014148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:16.014209 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:16.014282 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:16.014959 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.015698 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.016863 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.017271 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.017368 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:16.021016 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:47051: BlockingRecv error: recv got EOF from 127.27.186.130:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 15:02:16 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:16 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:16.461378 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:16.453695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:49587)
0809 15:02:16.454192 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:16.454226 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:16.454337 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:16.454593 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:16.454595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:16.454967 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:16.455117 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:16.455127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.456242 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.456246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:16.457298 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:16.457313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.457933 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.457936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:16.458096 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:16.458661 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:16.458678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:16.461137 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":248,"threads_started":1}
W20260809 15:02:16.461539 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:16.453732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:49587 (local address 127.27.186.190:44001)
0809 15:02:16.453897 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:16.453902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:16.454291 (+ 389us) server_negotiation.cc:407] Connection header received
0809 15:02:16.454421 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:16.454424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:16.454485 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:16.454560 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:16.455260 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.456107 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.457417 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.457826 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.457917 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:16.461322 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:49587: BlockingRecv error: recv got EOF from 127.27.186.131:49587 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 15:02:16 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:16 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:16.588670 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:16.580873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:48117)
0809 15:02:16.581364 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:16.581378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:16.581502 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:16.581822 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:16.581825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:16.582282 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:16.582443 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:16.582454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.583706 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.583710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:16.584605 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:16.584618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.585216 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.585220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:16.585333 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:16.585833 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:16.585849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:16.588417 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":235,"threads_started":1}
W20260809 15:02:16.588832 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:16.580958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:48117 (local address 127.27.186.190:44001)
0809 15:02:16.581192 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:16.581198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:16.581460 (+ 262us) server_negotiation.cc:407] Connection header received
0809 15:02:16.581614 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:16.581619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:16.581685 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:16.581774 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:16.582586 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.583579 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.584711 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:16.585115 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:16.585202 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:16.588599 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:48117: BlockingRecv error: recv got EOF from 127.27.186.129:48117 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 15:02:17 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:17 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:17.029597 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:17.021914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:32997)
0809 15:02:17.022464 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:17.022480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:17.022588 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:17.022848 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:17.022852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:17.023248 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:17.023407 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:17.023416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.024428 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.024432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:17.025375 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:17.025387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.026066 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.026074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:17.026224 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:17.026772 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:17.026790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:17.029352 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":290,"threads_started":1}
W20260809 15:02:17.029738 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:17.022089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:32997 (local address 127.27.186.190:44001)
0809 15:02:17.022237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:17.022241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:17.022542 (+ 301us) server_negotiation.cc:407] Connection header received
0809 15:02:17.022678 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:17.022682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:17.022739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:17.022819 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:17.023512 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.024254 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.025483 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.025878 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.025966 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:17.029530 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:32997: BlockingRecv error: recv got EOF from 127.27.186.130:32997 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 15:02:17 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:17 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:17.470041 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:17.462238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:35903)
0809 15:02:17.462750 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:17.462770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:17.462878 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:17.463151 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:17.463155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:17.463529 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:17.463704 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:17.463714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.464743 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.464748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:17.465659 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:17.465673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.466409 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.466414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:17.466533 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:17.467101 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:17.467117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:17.469782 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":248,"threads_started":1}
W20260809 15:02:17.470273 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:17.462290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:35903 (local address 127.27.186.190:44001)
0809 15:02:17.462488 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:17.462492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:17.462836 (+ 344us) server_negotiation.cc:407] Connection header received
0809 15:02:17.462970 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:17.462974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:17.463038 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:17.463132 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:17.463830 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.464558 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.465821 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.466289 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.466394 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:17.469964 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:35903: BlockingRecv error: recv got EOF from 127.27.186.131:35903 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 15:02:17 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:17 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:17.596956 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:17.589522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:43477)
0809 15:02:17.590002 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:17.590034 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:17.590143 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:17.590470 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:17.590473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:17.590827 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:17.590991 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:17.591000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.591997 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.592002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:17.592891 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:17.592904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.593593 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.593598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:17.593720 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:17.594266 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:17.594283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:17.596748 (+ 2465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
W20260809 15:02:17.597143 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:17.589585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:43477 (local address 127.27.186.190:44001)
0809 15:02:17.589734 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:17.589738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:17.590096 (+ 358us) server_negotiation.cc:407] Connection header received
0809 15:02:17.590252 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:17.590261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:17.590352 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:17.590463 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:17.591102 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.591864 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.593041 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:17.593464 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:17.593557 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:17.596931 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:43477: BlockingRecv error: recv got EOF from 127.27.186.129:43477 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 15:02:18 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:18 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:18.038262 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:18.030583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:34795)
0809 15:02:18.031079 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:18.031096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:18.031220 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:18.031491 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:18.031496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:18.031946 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:18.032102 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:18.032113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.033020 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.033023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:18.033908 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:18.033920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.034580 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.034585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:18.034757 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:18.035362 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:18.035379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:18.038038 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":268,"threads_started":1}
W20260809 15:02:18.038478 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:18.030655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:34795 (local address 127.27.186.190:44001)
0809 15:02:18.030822 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:18.030826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:18.031158 (+ 332us) server_negotiation.cc:407] Connection header received
0809 15:02:18.031317 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:18.031320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:18.031377 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:18.031452 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:18.032225 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.032916 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.034027 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.034441 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.034552 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:18.038243 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:34795: BlockingRecv error: recv got EOF from 127.27.186.130:34795 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 15:02:18 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:18 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:18.478801 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:18.470941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:36099)
0809 15:02:18.471485 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:18.471512 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:18.471642 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:18.471972 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:18.471976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:18.472379 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:18.472534 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:18.472545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.473575 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.473581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:18.474492 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:18.474505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.475249 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.475255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:18.475422 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:18.475924 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:18.475940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:18.478577 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":309,"threads_started":1}
W20260809 15:02:18.478988 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:18.471012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:36099 (local address 127.27.186.190:44001)
0809 15:02:18.471224 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:18.471230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:18.471579 (+ 349us) server_negotiation.cc:407] Connection header received
0809 15:02:18.471767 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:18.471772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:18.471833 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:18.471911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:18.472678 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.473436 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.474634 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.475088 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.475197 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:18.478760 (+ 3563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:36099: BlockingRecv error: recv got EOF from 127.27.186.131:36099 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 15:02:18 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:18 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:18.605163 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:18.597784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:56389)
0809 15:02:18.598333 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:18.598348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:18.598456 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:18.598691 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:18.598695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:18.599071 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:18.599233 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:18.599243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.600126 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.600129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:18.601075 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:18.601089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.601708 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.601711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:18.601827 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:18.602320 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:18.602339 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:18.604921 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":231,"threads_started":1}
W20260809 15:02:18.605252 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:18.597872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:56389 (local address 127.27.186.190:44001)
0809 15:02:18.598078 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:18.598084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:18.598415 (+ 331us) server_negotiation.cc:407] Connection header received
0809 15:02:18.598533 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:18.598536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:18.598593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:18.598674 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:18.599337 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.600014 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.601187 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:18.601592 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:18.601687 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:18.605090 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:56389: BlockingRecv error: recv got EOF from 127.27.186.129:56389 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 15:02:19 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:19 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:19.052074 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:19.039108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:45511)
0809 15:02:19.039609 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:19.039625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:19.039747 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:19.040017 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:19.040020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:19.040433 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:19.040600 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:19.040610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.041690 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.041694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:19.042649 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:19.042662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.043347 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.043352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:19.043478 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:19.048930 (+ 5452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:19.048948 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:19.051825 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":216,"threads_started":1}
W20260809 15:02:19.052294 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:19.039150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:45511 (local address 127.27.186.190:44001)
0809 15:02:19.039338 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:19.039346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:19.039693 (+ 347us) server_negotiation.cc:407] Connection header received
0809 15:02:19.039836 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:19.039840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:19.039912 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:19.039990 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:19.040745 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.041549 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.042763 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.043190 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.043282 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:19.052037 (+ 8755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:45511: BlockingRecv error: recv got EOF from 127.27.186.130:45511 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 15:02:19 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:19 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:19.487916 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:19.479670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:45231)
0809 15:02:19.480210 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:19.480230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:19.480361 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:19.480691 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:19.480696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:19.481139 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:19.481308 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:19.481318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.482375 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.482381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:19.483294 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:19.483307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.484017 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.484023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:19.484178 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:19.484730 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:19.484747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:19.487578 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":232,"threads_started":1}
W20260809 15:02:19.488003 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:19.479753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:45231 (local address 127.27.186.190:44001)
0809 15:02:19.479907 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:19.479913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:19.480323 (+ 410us) server_negotiation.cc:407] Connection header received
0809 15:02:19.480475 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:19.480481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:19.480560 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:19.480640 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:19.481465 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.482234 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.483433 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.483873 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.483980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:19.487789 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:45231: BlockingRecv error: recv got EOF from 127.27.186.131:45231 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 15:02:19 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:19 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:19.614168 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:19.605993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:45421)
0809 15:02:19.606541 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:19.606556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:19.606691 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:19.607031 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:19.607037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:19.607467 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:19.607620 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:19.607630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.608629 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.608633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:19.609588 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:19.609603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.610440 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.610446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:19.610583 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:19.611132 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:19.611148 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:19.613901 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":280,"threads_started":1}
W20260809 15:02:19.614320 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:19.606183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:45421 (local address 127.27.186.190:44001)
0809 15:02:19.606375 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:19.606382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:19.606626 (+ 244us) server_negotiation.cc:407] Connection header received
0809 15:02:19.606834 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:19.606838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:19.606898 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:19.606974 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:19.607757 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.608476 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.609722 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:19.610257 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:19.610365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:19.614097 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:45421: BlockingRecv error: recv got EOF from 127.27.186.129:45421 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 15:02:20 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:20 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:20.061113 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:20.052935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:44653)
0809 15:02:20.053410 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:20.053426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:20.053552 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:20.053878 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:20.053883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:20.054337 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:20.054490 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:20.054500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.055688 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.055692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:20.056585 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:20.056597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.057341 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.057347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:20.057489 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:20.058085 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:20.058103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:20.060794 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
W20260809 15:02:20.061199 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:20.052974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:44653 (local address 127.27.186.190:44001)
0809 15:02:20.053159 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:20.053166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:20.053504 (+ 338us) server_negotiation.cc:407] Connection header received
0809 15:02:20.053677 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:20.053683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:20.053751 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:20.053829 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:20.054632 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.055543 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.056732 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.057171 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.057277 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:20.060990 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:44653: BlockingRecv error: recv got EOF from 127.27.186.130:44653 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 15:02:20 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:20 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:20.497591 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:20.489047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:37515)
0809 15:02:20.489597 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:20.489617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:20.489725 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:20.489996 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:20.489999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:20.490413 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:20.490562 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:20.490572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.491632 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.491637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:20.492984 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:20.493001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.493926 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.493932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:20.494121 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:20.494699 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:20.494717 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:20.497284 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":323,"threads_started":1}
W20260809 15:02:20.497731 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:20.489102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37515 (local address 127.27.186.190:44001)
0809 15:02:20.489297 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:20.489304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:20.489688 (+ 384us) server_negotiation.cc:407] Connection header received
0809 15:02:20.489814 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:20.489817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:20.489891 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:20.489968 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:20.490678 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.491487 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.493111 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.493743 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.493865 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:20.497509 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:37515: BlockingRecv error: recv got EOF from 127.27.186.131:37515 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 15:02:20 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:20 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:20.622264 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:20.614947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:52069)
0809 15:02:20.615391 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:20.615407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:20.615515 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:20.615754 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:20.615758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:20.616123 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:20.616295 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:20.616306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.617200 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.617203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:20.618139 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:20.618152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.618756 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.618760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:20.618875 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:20.619410 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:20.619428 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:20.622006 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":218,"threads_started":1}
W20260809 15:02:20.622392 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:20.614986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:52069 (local address 127.27.186.190:44001)
0809 15:02:20.615174 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:20.615181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:20.615473 (+ 292us) server_negotiation.cc:407] Connection header received
0809 15:02:20.615593 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:20.615596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:20.615653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:20.615727 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:20.616397 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.617101 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.618244 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:20.618649 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:20.618735 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:20.622197 (+ 3462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:52069: BlockingRecv error: recv got EOF from 127.27.186.129:52069 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 15:02:21 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:21 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:21.069692 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:21.062117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:56719)
0809 15:02:21.062576 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:21.062592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:21.062711 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:21.062984 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:21.062987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:21.063369 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:21.063536 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:21.063545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.064452 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.064455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:21.065376 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:21.065386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.065987 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.065991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:21.066134 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:21.066626 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:21.066650 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:21.069386 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
W20260809 15:02:21.069787 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:21.062175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56719 (local address 127.27.186.190:44001)
0809 15:02:21.062368 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:21.062375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:21.062663 (+ 288us) server_negotiation.cc:407] Connection header received
0809 15:02:21.062812 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:21.062816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:21.062876 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:21.062954 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:21.063637 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.064352 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.065477 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.065879 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.065964 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:21.069560 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:56719: BlockingRecv error: recv got EOF from 127.27.186.130:56719 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 15:02:21 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:21 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:21.506182 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:21.498441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:55507)
0809 15:02:21.498919 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:21.498940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:21.499052 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:21.499319 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:21.499322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:21.499704 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:21.499890 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:21.499900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.500886 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.500890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:21.501808 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:21.501820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.502525 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.502529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:21.502657 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:21.503161 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:21.503179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:21.505894 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
W20260809 15:02:21.506448 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:21.498492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:55507 (local address 127.27.186.190:44001)
0809 15:02:21.498695 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:21.498703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:21.499011 (+ 308us) server_negotiation.cc:407] Connection header received
0809 15:02:21.499134 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:21.499137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:21.499211 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:21.499289 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:21.499998 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.500772 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.501923 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.502408 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.502502 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:21.506154 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:55507: BlockingRecv error: recv got EOF from 127.27.186.131:55507 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 15:02:21 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:21 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:21.631810 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:21.623187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:56577)
0809 15:02:21.623668 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:21.623684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:21.623832 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:21.624126 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:21.624131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:21.624597 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:21.624748 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:21.624758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.625764 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.625770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:21.626770 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:21.626784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.627509 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.627514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:21.627671 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:21.628237 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:21.628255 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:21.631487 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
W20260809 15:02:21.632010 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:21.623281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:56577 (local address 127.27.186.190:44001)
0809 15:02:21.623454 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:21.623460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:21.623765 (+ 305us) server_negotiation.cc:407] Connection header received
0809 15:02:21.623939 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:21.623943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:21.624003 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:21.624081 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:21.624890 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.625619 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.626914 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:21.627365 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:21.627482 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:21.631764 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:56577: BlockingRecv error: recv got EOF from 127.27.186.129:56577 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 15:02:22 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:22 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:22.079201 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:22.070638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:33287)
0809 15:02:22.071144 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:22.071161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:22.071302 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:22.071662 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:22.071667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:22.072082 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:22.072240 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:22.072251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.073308 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.073312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:22.074472 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:22.074488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.075389 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.075396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:22.075549 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:22.076119 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:22.076138 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:22.078984 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
W20260809 15:02:22.079460 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:22.070754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:33287 (local address 127.27.186.190:44001)
0809 15:02:22.070954 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:22.070962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:22.071245 (+ 283us) server_negotiation.cc:407] Connection header received
0809 15:02:22.071446 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:22.071450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:22.071516 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:22.071602 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:22.072397 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.073153 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.074626 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.075210 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.075337 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:22.079189 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:33287: BlockingRecv error: recv got EOF from 127.27.186.130:33287 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 15:02:22 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:22 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:22.515525 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:22.507263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:33665)
0809 15:02:22.507735 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:22.507755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:22.507879 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:22.508196 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:22.508200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:22.508649 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:22.508800 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:22.508811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.509829 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.509835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:22.510875 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:22.510889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.511604 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.511610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:22.511766 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:22.512401 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:22.512419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:22.515197 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
W20260809 15:02:22.515643 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:22.507408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:33665 (local address 127.27.186.190:44001)
0809 15:02:22.507613 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:22.507620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:22.507831 (+ 211us) server_negotiation.cc:407] Connection header received
0809 15:02:22.507991 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:22.507995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:22.508067 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:22.508146 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:22.508941 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.509698 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.511011 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.511438 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.511547 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:22.515409 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:33665: BlockingRecv error: recv got EOF from 127.27.186.131:33665 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 15:02:22 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:22 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:22.640501 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:22.632612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:35221)
0809 15:02:22.633082 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:22.633097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:22.633232 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:22.633537 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:22.633541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:22.633931 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:22.634159 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:22.634171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.635178 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.635183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:22.636050 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:22.636062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.636746 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.636752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:22.636919 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:22.637425 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:22.637441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:22.640236 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":253,"threads_started":1}
W20260809 15:02:22.640606 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:22.632691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:35221 (local address 127.27.186.190:44001)
0809 15:02:22.632868 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:22.632875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:22.633177 (+ 302us) server_negotiation.cc:407] Connection header received
0809 15:02:22.633340 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:22.633346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:22.633406 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:22.633482 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:22.634304 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.635046 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.636190 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:22.636605 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:22.636720 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:22.640437 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:35221: BlockingRecv error: recv got EOF from 127.27.186.129:35221 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 15:02:23 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:23 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:23.092393 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:23.079960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:43217)
0809 15:02:23.080440 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:23.080455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:23.080580 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:23.080908 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:23.080912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:23.081327 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:23.081511 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:23.081522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.082566 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.082571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:23.083448 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:23.083461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.084155 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.084161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:23.084334 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:23.088953 (+ 4619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:23.088971 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:23.092053 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":255,"threads_started":1}
W20260809 15:02:23.092593 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:23.080058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:43217 (local address 127.27.186.190:44001)
0809 15:02:23.080238 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:23.080245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:23.080534 (+ 289us) server_negotiation.cc:407] Connection header received
0809 15:02:23.080698 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:23.080704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:23.080765 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:23.080851 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:23.081654 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.082431 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.083585 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.084010 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.084115 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:23.092319 (+ 8204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:43217: BlockingRecv error: recv got EOF from 127.27.186.130:43217 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 15:02:23 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:23 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:23.523950 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:23.516370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:33461)
0809 15:02:23.516836 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:23.516857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:23.516964 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:23.517252 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:23.517255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:23.517646 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:23.517816 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:23.517825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.518818 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.518821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:23.519767 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:23.519780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.520378 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.520382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:23.520526 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:23.521107 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:23.521124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:23.523714 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
W20260809 15:02:23.524116 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:23.516431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:33461 (local address 127.27.186.190:44001)
0809 15:02:23.516576 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:23.516580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:23.516916 (+ 336us) server_negotiation.cc:407] Connection header received
0809 15:02:23.517060 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:23.517064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:23.517151 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:23.517226 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:23.517956 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.518720 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.519871 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.520267 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.520358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:23.523894 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:33461: BlockingRecv error: recv got EOF from 127.27.186.131:33461 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 15:02:23 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:23 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:23.648953 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:23.641454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:52835)
0809 15:02:23.641918 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:23.641933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:23.642075 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:23.642348 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:23.642350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:23.642719 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:23.642870 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:23.642880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.643795 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.643798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:23.644704 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:23.644716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.645341 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.645345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:23.645477 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:23.645987 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:23.646004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:23.648631 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
W20260809 15:02:23.649073 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:23.641440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:52835 (local address 127.27.186.190:44001)
0809 15:02:23.641619 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:23.641626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:23.642003 (+ 377us) server_negotiation.cc:407] Connection header received
0809 15:02:23.642177 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:23.642180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:23.642242 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:23.642319 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:23.642977 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.643686 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.644810 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:23.645241 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:23.645327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:23.648865 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:52835: BlockingRecv error: recv got EOF from 127.27.186.129:52835 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 15:02:24 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:24 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:24.105892 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:24.093739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:54421)
0809 15:02:24.094313 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:24.094328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:24.094457 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:24.094783 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:24.094787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:24.095197 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:24.095353 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:24.095363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.096321 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.096326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:24.097231 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:24.097243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.097964 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.097970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:24.098159 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:24.102842 (+ 4683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:24.102860 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:24.105539 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":347,"threads_started":1}
W20260809 15:02:24.105986 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:24.093848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:54421 (local address 127.27.186.190:44001)
0809 15:02:24.094138 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:24.094144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:24.094408 (+ 264us) server_negotiation.cc:407] Connection header received
0809 15:02:24.094575 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:24.094579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:24.094641 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:24.094732 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:24.095489 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.096179 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.097364 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.097811 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.097919 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:24.105768 (+ 7849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:54421: BlockingRecv error: recv got EOF from 127.27.186.130:54421 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 09 15:02:24 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:24 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:24.532397 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:24.524735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:51719)
0809 15:02:24.525157 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:24.525177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:24.525300 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:24.525574 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:24.525577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:24.525938 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:24.526145 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:24.526156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.527099 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.527103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:24.528132 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:24.528145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.528857 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.528863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:24.529022 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:24.529559 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:24.529576 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:24.532172 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":230,"threads_started":1}
W20260809 15:02:24.532557 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:24.524781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:51719 (local address 127.27.186.190:44001)
0809 15:02:24.524969 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:24.524976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:24.525261 (+ 285us) server_negotiation.cc:407] Connection header received
0809 15:02:24.525391 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:24.525395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:24.525466 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:24.525540 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:24.526256 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.526953 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.528279 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.528687 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.528806 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:24.532344 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:51719: BlockingRecv error: recv got EOF from 127.27.186.131:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 15:02:24 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:24 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:24.657773 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:24.649826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:36443)
0809 15:02:24.650317 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:24.650334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:24.650446 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:24.650779 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:24.650783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:24.651178 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:24.651373 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:24.651386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.652405 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.652409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:24.653328 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:24.653341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.654133 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.654139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:24.654305 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:24.654808 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:24.654835 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:24.657540 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
W20260809 15:02:24.657927 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:24.649902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:36443 (local address 127.27.186.190:44001)
0809 15:02:24.650125 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:24.650130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:24.650402 (+ 272us) server_negotiation.cc:407] Connection header received
0809 15:02:24.650574 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:24.650577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:24.650638 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:24.650733 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:24.651509 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.652270 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.653473 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:24.653938 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:24.654088 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:24.657722 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:36443: BlockingRecv error: recv got EOF from 127.27.186.129:36443 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 15:02:25 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:25 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:25.115181 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:25.106799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:50747)
0809 15:02:25.107309 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:25.107325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:25.107450 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:25.107787 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:25.107793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:25.108213 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:25.108376 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:25.108387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.109469 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.109475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:25.110469 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:25.110490 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.111217 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.111224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:25.111380 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:25.112011 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:25.112029 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:25.114952 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":250,"threads_started":1}
W20260809 15:02:25.115388 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:25.106967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:50747 (local address 127.27.186.190:44001)
0809 15:02:25.107209 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:25.107214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:25.107393 (+ 179us) server_negotiation.cc:407] Connection header received
0809 15:02:25.107564 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:25.107570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:25.107640 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:25.107722 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:25.108512 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.109330 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.110620 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.111045 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.111154 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:25.115169 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:50747: BlockingRecv error: recv got EOF from 127.27.186.130:50747 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 09 15:02:25 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:25 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:25.541237 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:25.533163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:37589)
0809 15:02:25.533703 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:25.533721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:25.533829 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:25.534243 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:25.534246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:25.534625 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:25.534784 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:25.534794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.535780 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.535784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:25.536722 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:25.536735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.537381 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.537386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:25.537509 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:25.538188 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:25.538211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:25.540987 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":285,"threads_started":1}
W20260809 15:02:25.541365 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:25.533266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37589 (local address 127.27.186.190:44001)
0809 15:02:25.533419 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:25.533423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:25.533782 (+ 359us) server_negotiation.cc:407] Connection header received
0809 15:02:25.533912 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:25.533915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:25.533974 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:25.534076 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:25.534921 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.535616 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.536836 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.537243 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.537330 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:25.541189 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:37589: BlockingRecv error: recv got EOF from 127.27.186.131:37589 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 15:02:25 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:25 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:25.666477 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:25.658550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:44629)
0809 15:02:25.658971 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:25.658986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:25.659096 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:25.659357 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:25.659360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:25.659697 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:25.659881 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:25.659891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.660852 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.660856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:25.661744 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:25.661755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.662376 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.662380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:25.662498 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:25.663087 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:25.663104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:25.666108 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
W20260809 15:02:25.666565 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:25.658604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:44629 (local address 127.27.186.190:44001)
0809 15:02:25.658910 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:25.658916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:25.659049 (+ 133us) server_negotiation.cc:407] Connection header received
0809 15:02:25.659184 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:25.659187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:25.659247 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:25.659325 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:25.659991 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.660743 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.661844 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:25.662274 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:25.662362 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:25.666325 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:44629: BlockingRecv error: recv got EOF from 127.27.186.129:44629 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Aug 09 15:02:26 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:26 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:26.123939 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:26.116037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:52749)
0809 15:02:26.116483 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:26.116498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:26.116634 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:26.117103 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:26.117107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:26.117462 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:26.117648 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:26.117660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.118724 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.118728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:26.119633 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:26.119645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.120326 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.120330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:26.120449 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:26.120986 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:26.121002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:26.123697 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
W20260809 15:02:26.124066 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:26.116170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:52749 (local address 127.27.186.190:44001)
0809 15:02:26.116326 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:26.116330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:26.116584 (+ 254us) server_negotiation.cc:407] Connection header received
0809 15:02:26.116760 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:26.116766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:26.116828 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:26.116910 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:26.117790 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.118593 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.119781 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.120216 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.120309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:26.123875 (+ 3566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:52749: BlockingRecv error: recv got EOF from 127.27.186.130:52749 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 15:02:26 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:26 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.131@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:26.550046 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:26.542079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.131:41669)
0809 15:02:26.542573 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:26.542593 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:26.542699 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:26.542981 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:26.542985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:26.543350 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:26.543520 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:26.543530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.544688 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.544692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:26.545785 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:26.545800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.546433 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.546436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:26.546549 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:26.547066 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:26.547083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:26.549705 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":243,"threads_started":1}
W20260809 15:02:26.550207 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:26.542135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:41669 (local address 127.27.186.190:44001)
0809 15:02:26.542283 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:26.542288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:26.542650 (+ 362us) server_negotiation.cc:407] Connection header received
0809 15:02:26.542799 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:26.542802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:26.542878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:26.542951 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:26.543625 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.544555 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.545912 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.546329 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.546421 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:26.549964 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.131:41669: BlockingRecv error: recv got EOF from 127.27.186.131:41669 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 15:02:26 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:26 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.129@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:26.675236 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:26.667498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.129:57729)
0809 15:02:26.667989 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:26.668004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:26.668129 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:26.668425 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:26.668430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:26.668889 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:26.669076 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:26.669086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.670089 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.670093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:26.670989 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:26.671002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.671724 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.671730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:26.671891 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:26.672393 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:26.672410 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:26.674990 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
W20260809 15:02:26.675367 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:26.667567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:57729 (local address 127.27.186.190:44001)
0809 15:02:26.667741 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:26.667747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:26.668082 (+ 335us) server_negotiation.cc:407] Connection header received
0809 15:02:26.668236 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:26.668241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:26.668301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:26.668375 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:26.669211 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.669929 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.671128 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:26.671589 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:26.671696 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:26.675179 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.129:57729: BlockingRecv error: recv got EOF from 127.27.186.129:57729 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 15:02:27 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 15:02:27 dist-test-slave-q9h9 krb5kdc[9337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.186.130@KRBTEST.COM for oryx/127.27.186.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 15:02:27.132998 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:27.124967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44001 (local address 127.27.186.130:51011)
0809 15:02:27.125477 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:27.125492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:27.125616 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:27.125944 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:27.125949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:27.126411 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:27.126640 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:27.126654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.127657 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.127662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:27.128795 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:27.128814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.129504 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.129510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:27.129652 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:27.130248 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:27.130266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:27.132753 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.186.190:44001: Server oryx/127.27.186.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
W20260809 15:02:27.133129 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:27.125033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:51011 (local address 127.27.186.190:44001)
0809 15:02:27.125225 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:27.125231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:27.125567 (+ 336us) server_negotiation.cc:407] Connection header received
0809 15:02:27.125748 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:27.125752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:27.125812 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:27.125887 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:27.126781 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.127511 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.128922 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.129387 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.129496 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:27.132926 (+ 3430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.186.130:51011: BlockingRecv error: recv got EOF from 127.27.186.130:51011 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:02:27.404155 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10315
I20260809 15:02:27.415791 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:27.405119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.27.186.129:46845)
0809 15:02:27.405631 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:27.405638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:27.405664 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:02:27.405725 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:27.405732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:27.405965 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:27.406179 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:27.407658 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.408756 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.409987 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.410565 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.410681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:27.412939 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:27.412947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:27.412958 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:27.412993 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:27.415210 (+ 2217us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.186.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 15:02:27.415494 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41688: Request ticket server kudu/127.27.186.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":237,"threads_started":1}
W20260809 15:02:27.415889 11073 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41688: Request ticket server kudu/127.27.186.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 15:02:27.415851 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10315: Not authorized: Client connection negotiation failed: client connection to 127.27.186.129:46845: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.186.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 15:02:27.416044 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 10315
I20260809 15:02:27.438787 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10572
I20260809 15:02:27.449965 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:27.439716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.27.186.130:33099)
0809 15:02:27.440281 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:27.440287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:27.440316 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:02:27.440372 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:27.440378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:27.440597 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:27.440745 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:27.441899 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.442961 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.444124 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.444635 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.444726 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:27.447134 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:27.447142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:27.447154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:27.447185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:27.449482 (+ 2297us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.186.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 15:02:27.449706 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48084: Request ticket server kudu/127.27.186.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":263,"threads_started":1}
W20260809 15:02:27.450083 11074 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48084: Request ticket server kudu/127.27.186.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 15:02:27.450098 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10572: Not authorized: Client connection negotiation failed: client connection to 127.27.186.130:33099: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.186.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 15:02:27.450248 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 10572
I20260809 15:02:27.470140 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10788
I20260809 15:02:27.481070 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:27.470934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.27.186.131:44393)
0809 15:02:27.471452 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:27.471459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:27.471484 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:02:27.471539 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:27.471546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:27.471744 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:27.471898 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:27.473162 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.474279 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.475454 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:27.475979 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:27.476089 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:27.478361 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:27.478369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:27.478381 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:27.478411 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:27.480493 (+ 2082us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.186.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 15:02:27.480769 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34602: Request ticket server kudu/127.27.186.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":245,"threads_started":1}
W20260809 15:02:27.481078 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10788: Not authorized: Client connection negotiation failed: client connection to 127.27.186.131:44393: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.186.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 15:02:27.481213 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 10788
W20260809 15:02:27.481194 11075 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34602: Request ticket server kudu/127.27.186.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 15:02:27.499790 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 10206
I20260809 15:02:27.500767 10301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:27.613891 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 10206
2026-08-09T15:02:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20919 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T15:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:27Z Disabled control of system clock
I20260809 15:02:27.690985 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:36151
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 15:02:27.996295 11083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:27.996539 11083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:27.996636 11083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:28.003407 11083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:28.003491 11083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:28.003533 11083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:28.003609 11083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:28.003664 11083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:28.012125 11083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:36151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:28.013197 11083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:28.014822 11083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:28.025501 11097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:28.025621 11092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:28.025643 11090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:28.025869 11083 server_base.cc:1034] running on GCE node
I20260809 15:02:28.026852 11083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:28.028045 11083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:28.029256 11083 hybrid_clock.cc:648] HybridClock initialized: now 1786287748029202 us; error 44 us; skew 500 ppm
I20260809 15:02:28.029565 11083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:28.031622 11083 webserver.cc:460] Webserver started at http://127.27.186.190:39727/ using document root <none> and password file <none>
I20260809 15:02:28.032235 11083 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:28.032335 11083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:28.032637 11083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:28.035290 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "a70ae8250d904170ad5a4bd7b3db17fe"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "b63d80c1f7b2368c332c9b5412fc2bc2"
server_key_iv: "5f13970f0b9fa153cd951b7f88f9b9fd"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.035991 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "a70ae8250d904170ad5a4bd7b3db17fe"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "b63d80c1f7b2368c332c9b5412fc2bc2"
server_key_iv: "5f13970f0b9fa153cd951b7f88f9b9fd"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.040587 11083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 15:02:28.043781 11111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:28.044960 11083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:02:28.045110 11083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "a70ae8250d904170ad5a4bd7b3db17fe"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "b63d80c1f7b2368c332c9b5412fc2bc2"
server_key_iv: "5f13970f0b9fa153cd951b7f88f9b9fd"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.045291 11083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:28.059871 11083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:28.060526 11083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:28.060768 11083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:28.077258 11083 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:36151
I20260809 15:02:28.077310 11200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:36151 every 8 connection(s)
I20260809 15:02:28.078426 11083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:02:28.081722 11201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:28.088498 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 11083
I20260809 15:02:28.088685 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:02:28.089470 28394 external_mini_cluster.cc:1428] Setting key 9c17aaebdd981ca61906b17e38d601e8
I20260809 15:02:28.092007 11201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe: Bootstrap starting.
I20260809 15:02:28.094990 11201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:28.096066 11201 log.cc:824] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:28.098577 11201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe: No bootstrap required, opened a new log
I20260809 15:02:28.102846 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:28.091430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.27.186.190:36151)
0809 15:02:28.092316 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:28.092326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:28.092365 (+ 39us) server_negotiation.cc:407] Connection header received
0809 15:02:28.094011 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:28.094062 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:28.094450 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:28.094739 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:28.095435 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:28.096735 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:28.098011 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:28.098684 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:28.098835 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:28.100052 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:28.100063 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:28.100079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:28.100666 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:28.100736 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:28.100755 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:28.100862 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:28.101306 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 15:02:28.101614 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":321,"threads_started":1}
I20260809 15:02:28.107306 11201 raft_consensus.cc:348] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36151 } }
I20260809 15:02:28.107599 11201 raft_consensus.cc:374] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:28.107671 11201 raft_consensus.cc:729] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70ae8250d904170ad5a4bd7b3db17fe, State: Initialized, Role: FOLLOWER
I20260809 15:02:28.108155 11201 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36151 } }
I20260809 15:02:28.108358 11201 raft_consensus.cc:388] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:02:28.108455 11201 raft_consensus.cc:482] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:02:28.108569 11201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:28.110567 11201 raft_consensus.cc:504] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36151 } }
I20260809 15:02:28.110972 11201 leader_election.cc:302] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a70ae8250d904170ad5a4bd7b3db17fe; no voters:
I20260809 15:02:28.111570 11201 leader_election.cc:288] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:02:28.111708 11210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:28.112376 11210 raft_consensus.cc:686] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [term 1 LEADER]: Becoming Leader. State: Replica: a70ae8250d904170ad5a4bd7b3db17fe, State: Running, Role: LEADER
I20260809 15:02:28.112901 11210 consensus_queue.cc:237] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36151 } }
I20260809 15:02:28.113477 11201 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:02:28.115520 11214 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70ae8250d904170ad5a4bd7b3db17fe. Latest consensus state: current_term: 1 leader_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36151 } } }
I20260809 15:02:28.115594 11213 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70ae8250d904170ad5a4bd7b3db17fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 36151 } } }
I20260809 15:02:28.115928 11214 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:28.115958 11213 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:28.117455 11222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:02:28.122282 11222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:02:28.130502 11222 catalog_manager.cc:1347] Generated new cluster ID: 2433d3038e2e4b1aaa4691dd5ab7373d
I20260809 15:02:28.130596 11222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:02:28.156507 11222 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:02:28.157636 11222 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:02:28.163388 11222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a70ae8250d904170ad5a4bd7b3db17fe: Generated new TSK 0
I20260809 15:02:28.164243 11222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:02:28.175598 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36151
--builtin_ntp_servers=127.27.186.148:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 15:02:28.496862 11237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:28.497095 11237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:28.497223 11237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:28.504096 11237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:28.504181 11237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:28.504311 11237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:02:28.512811 11237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:28.513809 11237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:28.515746 11237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:28.528317 11249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:28.528258 11251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:28.528347 11248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:28.529083 11237 server_base.cc:1034] running on GCE node
I20260809 15:02:28.529619 11237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:28.530489 11237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:28.531674 11237 hybrid_clock.cc:648] HybridClock initialized: now 1786287748531626 us; error 40 us; skew 500 ppm
I20260809 15:02:28.531945 11237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:28.533896 11237 webserver.cc:460] Webserver started at http://127.27.186.129:33129/ using document root <none> and password file <none>
I20260809 15:02:28.534536 11237 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:28.534627 11237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:28.534932 11237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:28.537429 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "1997fa08bda64e32bddd8b8663f0860a"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "5e4a956233872c65f08fc9951294e7dc"
server_key_iv: "467f1dae1aed2d89dfe6fcdfe7fcd72d"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.538206 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "1997fa08bda64e32bddd8b8663f0860a"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "5e4a956233872c65f08fc9951294e7dc"
server_key_iv: "467f1dae1aed2d89dfe6fcdfe7fcd72d"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.542779 11237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:02:28.545835 11258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:28.546985 11237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:02:28.547143 11237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "1997fa08bda64e32bddd8b8663f0860a"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "5e4a956233872c65f08fc9951294e7dc"
server_key_iv: "467f1dae1aed2d89dfe6fcdfe7fcd72d"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.547324 11237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:28.563580 11237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:28.564323 11237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:28.564553 11237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:28.565303 11237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:28.566531 11237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:28.566613 11237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:28.566689 11237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:28.566740 11237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:28.596465 11237 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:44907
I20260809 15:02:28.596522 11441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:44907 every 8 connection(s)
I20260809 15:02:28.597647 11237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:02:28.600811 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 11237
I20260809 15:02:28.600980 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:02:28.601775 28394 external_mini_cluster.cc:1428] Setting key 7460bf4819ad064fdaa5e3bf38becdf6
I20260809 15:02:28.604635 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36151
--builtin_ntp_servers=127.27.186.148:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 15:02:28.609858 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:28.599812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:35643 (local address 127.27.186.190:36151)
0809 15:02:28.600017 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:28.600023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:28.601403 (+ 1380us) server_negotiation.cc:407] Connection header received
0809 15:02:28.603192 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:28.603198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:28.603273 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:28.603361 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:28.604602 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:28.605370 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:28.607417 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:28.607804 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:28.607895 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:28.609220 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:28.609225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:28.609228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:28.609245 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:28.609275 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:28.609286 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:28.609348 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:28.609557 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 15:02:28.609631 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:02:28.610982 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:28.600285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36151 (local address 127.27.186.129:35643)
0809 15:02:28.601240 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:28.601301 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:28.602946 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:28.603587 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:28.603598 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:28.603699 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:28.604419 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:28.604445 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:28.605537 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:28.605542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:28.607295 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:28.607309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:28.608001 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:28.608007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:28.608243 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:28.608892 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:02:28.608896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:28.608938 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:02:28.609023 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:28.609370 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:28.609380 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:28.609386 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:02:28.609549 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 15:02:28.609865 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":304,"threads_started":1}
I20260809 15:02:28.612308 11442 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36151
I20260809 15:02:28.612599 11442 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:28.613418 11442 heartbeater.cc:511] Master 127.27.186.190:36151 requested a full tablet report, sending...
I20260809 15:02:28.615556 11138 ts_manager.cc:194] Registered new tserver with Master: 1997fa08bda64e32bddd8b8663f0860a (127.27.186.129:44907)
I20260809 15:02:28.617115 11138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.129:35643
W20260809 15:02:28.918941 11446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:28.919202 11446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:28.919327 11446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:28.926173 11446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:28.926273 11446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:28.926405 11446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:02:28.934845 11446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:28.935891 11446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:28.937496 11446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:28.949673 11454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:28.949717 11456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:28.949740 11453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:28.950704 11446 server_base.cc:1034] running on GCE node
I20260809 15:02:28.951236 11446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:28.952095 11446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:28.953293 11446 hybrid_clock.cc:648] HybridClock initialized: now 1786287748953239 us; error 41 us; skew 500 ppm
I20260809 15:02:28.953574 11446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:28.955592 11446 webserver.cc:460] Webserver started at http://127.27.186.130:40029/ using document root <none> and password file <none>
I20260809 15:02:28.956199 11446 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:28.956300 11446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:28.956596 11446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:28.959213 11446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "bcfb104477c34edd9b58fa1722bf520b"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "ad6abeea42c323050b5b48a5bf484afc"
server_key_iv: "d03c820f6beeed5079a708e0a4c51a20"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.959939 11446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "bcfb104477c34edd9b58fa1722bf520b"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "ad6abeea42c323050b5b48a5bf484afc"
server_key_iv: "d03c820f6beeed5079a708e0a4c51a20"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.964623 11446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:02:28.967935 11464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:28.969126 11446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:02:28.969266 11446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "bcfb104477c34edd9b58fa1722bf520b"
format_stamp: "Formatted at 2026-08-09 15:02:28 on dist-test-slave-q9h9"
server_key: "ad6abeea42c323050b5b48a5bf484afc"
server_key_iv: "d03c820f6beeed5079a708e0a4c51a20"
server_key_version: "encryptionkey@0"
I20260809 15:02:28.969460 11446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:29.003571 11446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:29.004293 11446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:29.004552 11446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:29.005421 11446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:29.006738 11446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:29.006831 11446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:29.006930 11446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:29.007000 11446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:29.036201 11446 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:34277
I20260809 15:02:29.036265 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:34277 every 8 connection(s)
I20260809 15:02:29.037340 11446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:02:29.038865 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 11446
I20260809 15:02:29.039040 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:02:29.039554 28394 external_mini_cluster.cc:1428] Setting key 874094c068e9092f2171628f956260d6
I20260809 15:02:29.042456 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36151
--builtin_ntp_servers=127.27.186.148:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 15:02:29.049571 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.039849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:57247 (local address 127.27.186.190:36151)
0809 15:02:29.040051 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:29.040057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:29.041534 (+ 1477us) server_negotiation.cc:407] Connection header received
0809 15:02:29.043396 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:29.043403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:29.043478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:29.043583 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:29.044536 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.045448 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.047002 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.047522 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.047643 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.048855 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:29.048860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:29.048863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:29.048879 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:29.048911 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:29.048923 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:29.048985 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:29.049221 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 15:02:29.049297 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:02:29.050710 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.040368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36151 (local address 127.27.186.130:57247)
0809 15:02:29.041403 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:29.041450 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:29.043138 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:29.043768 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:29.043775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:29.043853 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:29.044369 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:29.044385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.045606 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.045611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:29.046864 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:29.046877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.047706 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.047711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:29.047892 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.048544 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:02:29.048547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:29.048592 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:02:29.048684 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:29.049021 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:29.049031 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:29.049038 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:02:29.049219 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 15:02:29.049546 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":315,"threads_started":1}
I20260809 15:02:29.051987 11658 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36151
I20260809 15:02:29.052263 11658 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:29.052922 11658 heartbeater.cc:511] Master 127.27.186.190:36151 requested a full tablet report, sending...
I20260809 15:02:29.054212 11138 ts_manager.cc:194] Registered new tserver with Master: bcfb104477c34edd9b58fa1722bf520b (127.27.186.130:34277)
I20260809 15:02:29.054967 11138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.130:57247
W20260809 15:02:29.362068 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:29.362316 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:29.362442 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:29.369207 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:29.369297 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:29.369428 11663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:02:29.377985 11663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:29.378998 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:29.380610 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:29.392230 11669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:29.392334 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:29.392769 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:29.393443 11663 server_base.cc:1034] running on GCE node
I20260809 15:02:29.393981 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:29.394850 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:29.396098 11663 hybrid_clock.cc:648] HybridClock initialized: now 1786287749396052 us; error 34 us; skew 500 ppm
I20260809 15:02:29.396413 11663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:29.401185 11663 webserver.cc:460] Webserver started at http://127.27.186.131:36137/ using document root <none> and password file <none>
I20260809 15:02:29.401793 11663 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:29.401885 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:29.402228 11663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:29.404798 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "df69c4c06a574e669a9014fb0b006ee8"
format_stamp: "Formatted at 2026-08-09 15:02:29 on dist-test-slave-q9h9"
server_key: "6a70cc1089c8a4aa97f2f22e417108e7"
server_key_iv: "69e42c13211b6ea968f00c630d567d68"
server_key_version: "encryptionkey@0"
I20260809 15:02:29.405565 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "df69c4c06a574e669a9014fb0b006ee8"
format_stamp: "Formatted at 2026-08-09 15:02:29 on dist-test-slave-q9h9"
server_key: "6a70cc1089c8a4aa97f2f22e417108e7"
server_key_iv: "69e42c13211b6ea968f00c630d567d68"
server_key_version: "encryptionkey@0"
I20260809 15:02:29.410245 11663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 15:02:29.413389 11685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:29.414609 11663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:02:29.414755 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "df69c4c06a574e669a9014fb0b006ee8"
format_stamp: "Formatted at 2026-08-09 15:02:29 on dist-test-slave-q9h9"
server_key: "6a70cc1089c8a4aa97f2f22e417108e7"
server_key_iv: "69e42c13211b6ea968f00c630d567d68"
server_key_version: "encryptionkey@0"
I20260809 15:02:29.414958 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:29.440083 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:29.440790 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:29.441046 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:29.441877 11663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:29.443192 11663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:29.443284 11663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:29.443379 11663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:29.443451 11663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:29.476574 11663 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:43859
I20260809 15:02:29.476643 11854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:43859 every 8 connection(s)
I20260809 15:02:29.477735 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:02:29.486282 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 11663
I20260809 15:02:29.486416 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:02:29.486840 28394 external_mini_cluster.cc:1428] Setting key 405ae63aa3e28e80bdd8d8046b5b22cd
I20260809 15:02:29.488808 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.479994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:33441 (local address 127.27.186.190:36151)
0809 15:02:29.480203 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:29.480209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:29.481427 (+ 1218us) server_negotiation.cc:407] Connection header received
0809 15:02:29.483060 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:29.483065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:29.483131 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:29.483215 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:29.484082 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.484764 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.486222 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.486702 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.486815 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.488024 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:29.488030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:29.488035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:29.488055 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:29.488097 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:29.488114 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:29.488187 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:29.488422 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 15:02:29.488503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:02:29.489880 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.480468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:36151 (local address 127.27.186.131:33441)
0809 15:02:29.481291 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:29.481331 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:29.482851 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:29.483377 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:29.483383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:29.483467 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:29.483958 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:29.483974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.484889 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.484892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:29.486075 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:29.486087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.486925 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.486933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:29.487145 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.487738 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:02:29.487742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:29.487779 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:02:29.487851 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:29.488219 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:29.488230 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:29.488237 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:02:29.488429 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 15:02:29.488750 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":249,"threads_started":1}
I20260809 15:02:29.491074 11855 heartbeater.cc:347] Connected to a master server at 127.27.186.190:36151
I20260809 15:02:29.491326 11855 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:29.491981 11855 heartbeater.cc:511] Master 127.27.186.190:36151 requested a full tablet report, sending...
I20260809 15:02:29.493192 11138 ts_manager.cc:194] Registered new tserver with Master: df69c4c06a574e669a9014fb0b006ee8 (127.27.186.131:43859)
I20260809 15:02:29.493908 11138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.131:33441
I20260809 15:02:29.501662 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 15:02:29.507442 11206 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:29.505779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.27.186.190:36151)
0809 15:02:29.505919 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:29.505922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:29.506453 (+ 531us) server_negotiation.cc:407] Connection header received
0809 15:02:29.506626 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:29.506630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:29.506697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:29.506779 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:29.507211 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37182: BlockingRecv error: recv got EOF from 127.0.0.1:37182 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:02:29.508153 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 11237
I20260809 15:02:29.514958 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.508971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.27.186.129:44907)
0809 15:02:29.509461 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:29.509469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:29.509492 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:02:29.509550 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:29.509556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:29.509716 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:29.509828 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:29.510166 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.511113 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.512247 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.512749 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.512840 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.513684 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:29.513691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:29.513701 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:29.514282 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:29.514330 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:29.514349 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:29.514433 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:29.514547 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 15:02:29.514677 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":274,"threads_started":1}
I20260809 15:02:29.516016 11432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:29.597005 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 11237
I20260809 15:02:29.622681 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 11446
I20260809 15:02:29.631559 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.623501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.27.186.130:34277)
0809 15:02:29.623978 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:29.623985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:29.624012 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:02:29.624077 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:29.624083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:29.624299 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:29.624429 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:29.624732 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.625819 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.627652 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.628485 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.628612 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.629760 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:29.629771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:29.629788 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:29.630631 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:29.630692 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:29.630733 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:29.630840 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:29.630998 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 15:02:29.631152 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":243,"threads_started":1}
I20260809 15:02:29.632813 11652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:29.710958 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 11446
I20260809 15:02:29.730916 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 11663
I20260809 15:02:29.738802 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:29.731813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.27.186.131:43859)
0809 15:02:29.732427 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:29.732433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:29.732462 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:02:29.732520 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:29.732526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:29.732754 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:29.732899 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:29.733358 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.734448 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.735711 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:29.736263 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:29.736387 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:29.737366 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:29.737374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:29.737386 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:29.737920 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:29.737964 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:29.737983 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:29.738226 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:29.738343 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 15:02:29.738467 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":315,"threads_started":1}
I20260809 15:02:29.739934 11847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:29.820695 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 11663
I20260809 15:02:29.841136 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 11083
I20260809 15:02:29.842239 11189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:29.941272 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 11083
2026-08-09T15:02:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2325 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T15:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:29Z Disabled control of system clock
I20260809 15:02:30.017424 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:34969
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:34969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 15:02:30.325765 11886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:30.326000 11886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:30.326138 11886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:30.332968 11886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:30.333055 11886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:30.333093 11886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:30.333181 11886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:30.333232 11886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:30.341725 11886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:34969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:34969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:30.342742 11886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:30.344333 11886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:30.354875 11893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:30.354921 11897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:30.354885 11894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:30.355739 11886 server_base.cc:1034] running on GCE node
I20260809 15:02:30.356371 11886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:30.357530 11886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:30.358726 11886 hybrid_clock.cc:648] HybridClock initialized: now 1786287750358669 us; error 46 us; skew 500 ppm
I20260809 15:02:30.359009 11886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:30.360929 11886 webserver.cc:460] Webserver started at http://127.27.186.190:43607/ using document root <none> and password file <none>
I20260809 15:02:30.361488 11886 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:30.361568 11886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:30.361828 11886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:30.364452 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "bf0ee9d6cb684e0681ea2ffd80e81707"
format_stamp: "Formatted at 2026-08-09 15:02:30 on dist-test-slave-q9h9"
server_key: "5b8916bea9016f8823628ecc60dc88e0"
server_key_iv: "47230efdaee6e026e55c8e8f4ce74819"
server_key_version: "encryptionkey@0"
I20260809 15:02:30.365129 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "bf0ee9d6cb684e0681ea2ffd80e81707"
format_stamp: "Formatted at 2026-08-09 15:02:30 on dist-test-slave-q9h9"
server_key: "5b8916bea9016f8823628ecc60dc88e0"
server_key_iv: "47230efdaee6e026e55c8e8f4ce74819"
server_key_version: "encryptionkey@0"
I20260809 15:02:30.369808 11886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:02:30.372844 11911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:30.374086 11886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 15:02:30.374214 11886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "bf0ee9d6cb684e0681ea2ffd80e81707"
format_stamp: "Formatted at 2026-08-09 15:02:30 on dist-test-slave-q9h9"
server_key: "5b8916bea9016f8823628ecc60dc88e0"
server_key_iv: "47230efdaee6e026e55c8e8f4ce74819"
server_key_version: "encryptionkey@0"
I20260809 15:02:30.374383 11886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:30.379434 11886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:30.380062 11886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:30.380295 11886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:30.396739 11886 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:34969
I20260809 15:02:30.396793 11991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:34969 every 8 connection(s)
I20260809 15:02:30.397898 11886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:02:30.401410 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:30.404412 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 11886
I20260809 15:02:30.404589 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:02:30.405212 28394 external_mini_cluster.cc:1428] Setting key 71a33c94832b45a20948a4e64af6a2ca
I20260809 15:02:30.412911 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:30.407064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.27.186.190:34969)
0809 15:02:30.408042 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:30.408052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:30.408093 (+ 41us) server_negotiation.cc:407] Connection header received
0809 15:02:30.409602 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:30.409623 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:30.410047 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:30.410430 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:30.411419 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:30.411425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:30.411437 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:30.411465 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:30.411521 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:30.411537 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:30.411742 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:30.411872 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 15:02:30.412093 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":365,"threads_started":1}
I20260809 15:02:30.413085 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707: Bootstrap starting.
I20260809 15:02:30.416277 11992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:30.417296 11992 log.cc:824] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:30.419757 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707: No bootstrap required, opened a new log
I20260809 15:02:30.426689 11992 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 34969 } }
I20260809 15:02:30.427013 11992 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:30.427100 11992 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0ee9d6cb684e0681ea2ffd80e81707, State: Initialized, Role: FOLLOWER
I20260809 15:02:30.427629 11992 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 34969 } }
I20260809 15:02:30.427860 11992 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:02:30.427978 11992 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:02:30.428121 11992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:30.430111 11992 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 34969 } }
I20260809 15:02:30.430545 11992 leader_election.cc:302] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0ee9d6cb684e0681ea2ffd80e81707; no voters:
I20260809 15:02:30.431113 11992 leader_election.cc:288] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:02:30.431263 11999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:30.432210 11999 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [term 1 LEADER]: Becoming Leader. State: Replica: bf0ee9d6cb684e0681ea2ffd80e81707, State: Running, Role: LEADER
I20260809 15:02:30.432734 11999 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 34969 } }
I20260809 15:02:30.433111 11992 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:02:30.435331 12000 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 34969 } } }
I20260809 15:02:30.435683 12000 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:30.436110 12001 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0ee9d6cb684e0681ea2ffd80e81707. Latest consensus state: current_term: 1 leader_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ee9d6cb684e0681ea2ffd80e81707" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 34969 } } }
I20260809 15:02:30.436362 12001 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707 [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:30.440064 12013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:02:30.444823 12013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:02:30.452760 12013 catalog_manager.cc:1347] Generated new cluster ID: dc7eb5f34a594146b7f17dbe961430d7
I20260809 15:02:30.452855 12013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:02:30.472110 12013 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:02:30.472445 12013 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:02:30.480572 12013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0ee9d6cb684e0681ea2ffd80e81707: Generated new TSK 0
I20260809 15:02:30.481253 12013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:02:30.495179 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:34969
--builtin_ntp_servers=127.27.186.148:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 15:02:30.847025 12029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:30.847257 12029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:30.847391 12029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:30.854221 12029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:30.854311 12029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:30.854456 12029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:02:30.862905 12029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:30.863927 12029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:30.865516 12029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:30.877410 12038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:30.877478 12039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:30.877440 12041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:30.878527 12029 server_base.cc:1034] running on GCE node
I20260809 15:02:30.879047 12029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:30.879863 12029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:30.881052 12029 hybrid_clock.cc:648] HybridClock initialized: now 1786287750881008 us; error 33 us; skew 500 ppm
I20260809 15:02:30.881341 12029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:30.883373 12029 webserver.cc:460] Webserver started at http://127.27.186.129:35459/ using document root <none> and password file <none>
I20260809 15:02:30.883998 12029 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:30.884093 12029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:30.884388 12029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:30.886977 12029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "026acffef3fc43659a8cec21b07d1fcc"
format_stamp: "Formatted at 2026-08-09 15:02:30 on dist-test-slave-q9h9"
server_key: "f7039a29863da840496ada9cd6d78f77"
server_key_iv: "4c8e0fbaea6bbbcc47cd2ece1040b83e"
server_key_version: "encryptionkey@0"
I20260809 15:02:30.887712 12029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "026acffef3fc43659a8cec21b07d1fcc"
format_stamp: "Formatted at 2026-08-09 15:02:30 on dist-test-slave-q9h9"
server_key: "f7039a29863da840496ada9cd6d78f77"
server_key_iv: "4c8e0fbaea6bbbcc47cd2ece1040b83e"
server_key_version: "encryptionkey@0"
I20260809 15:02:30.892310 12029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:02:30.895426 12047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:30.896579 12029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:02:30.896710 12029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "026acffef3fc43659a8cec21b07d1fcc"
format_stamp: "Formatted at 2026-08-09 15:02:30 on dist-test-slave-q9h9"
server_key: "f7039a29863da840496ada9cd6d78f77"
server_key_iv: "4c8e0fbaea6bbbcc47cd2ece1040b83e"
server_key_version: "encryptionkey@0"
I20260809 15:02:30.896906 12029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:30.902158 12029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:30.902855 12029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:30.903090 12029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:30.903895 12029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:30.905155 12029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:30.905243 12029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:30.905355 12029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:30.905432 12029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:30.935048 12029 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:46469
I20260809 15:02:30.935107 12230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:46469 every 8 connection(s)
I20260809 15:02:30.936187 12029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:02:30.943490 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:30.938448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:48775 (local address 127.27.186.190:34969)
0809 15:02:30.938902 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:30.938908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:30.940033 (+ 1125us) server_negotiation.cc:407] Connection header received
0809 15:02:30.941663 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:30.941667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:30.941734 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:30.941824 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:30.942939 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:30.942944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:30.942947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:30.942963 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:30.943005 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:30.943016 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:30.943107 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:30.943227 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 15:02:30.943273 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260809 15:02:30.943825 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 12029
I20260809 15:02:30.944033 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:02:30.944593 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:30.938980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:34969 (local address 127.27.186.129:48775)
0809 15:02:30.939903 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:30.939938 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:30.941449 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:30.941995 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:30.942002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:30.942125 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:30.942676 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:02:30.942678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:30.942711 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:02:30.942791 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:30.943087 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:30.943092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:30.943097 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:02:30.943294 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 15:02:30.943505 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":270,"threads_started":1}
I20260809 15:02:30.944845 28394 external_mini_cluster.cc:1428] Setting key dd29b003ac17826a6340f0b6fcfda55d
I20260809 15:02:30.946056 12232 heartbeater.cc:347] Connected to a master server at 127.27.186.190:34969
I20260809 15:02:30.946326 12232 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:30.946973 12232 heartbeater.cc:511] Master 127.27.186.190:34969 requested a full tablet report, sending...
I20260809 15:02:30.947796 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:34969
--builtin_ntp_servers=127.27.186.148:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 15:02:30.948949 11937 ts_manager.cc:194] Registered new tserver with Master: 026acffef3fc43659a8cec21b07d1fcc (127.27.186.129:46469)
W20260809 15:02:31.260664 12240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:31.260905 12240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:31.260989 12240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:31.267822 12240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:31.267903 12240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:31.268016 12240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:02:31.276510 12240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:31.277575 12240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:31.279218 12240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:31.291047 12248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:31.291152 12249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:31.291164 12253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:31.292083 12240 server_base.cc:1034] running on GCE node
I20260809 15:02:31.292601 12240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:31.293403 12240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:31.294579 12240 hybrid_clock.cc:648] HybridClock initialized: now 1786287751294526 us; error 49 us; skew 500 ppm
I20260809 15:02:31.294895 12240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:31.296905 12240 webserver.cc:460] Webserver started at http://127.27.186.130:33021/ using document root <none> and password file <none>
I20260809 15:02:31.297501 12240 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:31.297595 12240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:31.297900 12240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:31.300621 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "d9a684f973ff4f60bba67d32698ec575"
format_stamp: "Formatted at 2026-08-09 15:02:31 on dist-test-slave-q9h9"
server_key: "cee3accda86e4d0d7b1e9d8398ccc0ee"
server_key_iv: "b59908ca969fcad82c60f01538ddf4d2"
server_key_version: "encryptionkey@0"
I20260809 15:02:31.301374 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a684f973ff4f60bba67d32698ec575"
format_stamp: "Formatted at 2026-08-09 15:02:31 on dist-test-slave-q9h9"
server_key: "cee3accda86e4d0d7b1e9d8398ccc0ee"
server_key_iv: "b59908ca969fcad82c60f01538ddf4d2"
server_key_version: "encryptionkey@0"
I20260809 15:02:31.305997 12240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 15:02:31.309163 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:31.310371 12240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260809 15:02:31.310511 12240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "d9a684f973ff4f60bba67d32698ec575"
format_stamp: "Formatted at 2026-08-09 15:02:31 on dist-test-slave-q9h9"
server_key: "cee3accda86e4d0d7b1e9d8398ccc0ee"
server_key_iv: "b59908ca969fcad82c60f01538ddf4d2"
server_key_version: "encryptionkey@0"
I20260809 15:02:31.310706 12240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:31.315809 12240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:31.316497 12240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:31.316730 12240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:31.317543 12240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:31.318859 12240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:31.318953 12240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:31.319054 12240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:31.319123 12240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:31.348194 12240 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:38361
I20260809 15:02:31.348249 12440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:38361 every 8 connection(s)
I20260809 15:02:31.349323 12240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:02:31.350314 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 12240
I20260809 15:02:31.350497 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:02:31.351047 28394 external_mini_cluster.cc:1428] Setting key e4c986e7824467275134b7a9b2e6eac4
I20260809 15:02:31.354162 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:34969
--builtin_ntp_servers=127.27.186.148:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 15:02:31.357640 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:31.351793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:54855 (local address 127.27.186.190:34969)
0809 15:02:31.351993 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:31.351999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:31.353474 (+ 1475us) server_negotiation.cc:407] Connection header received
0809 15:02:31.355636 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:31.355640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:31.355689 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:31.355768 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:31.357075 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:31.357079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:31.357083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:31.357099 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:31.357132 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:31.357144 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:31.357233 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:31.357413 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 15:02:31.357463 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:02:31.358718 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:31.352344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:34969 (local address 127.27.186.130:54855)
0809 15:02:31.353317 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:31.353369 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:31.355378 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:31.355997 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:31.356010 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:31.356120 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:31.356758 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:02:31.356761 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:31.356806 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:02:31.356904 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:31.357224 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:31.357233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:31.357239 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:02:31.357480 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 15:02:31.357691 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":298,"threads_started":1}
I20260809 15:02:31.359929 12441 heartbeater.cc:347] Connected to a master server at 127.27.186.190:34969
I20260809 15:02:31.360208 12441 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:31.360805 12441 heartbeater.cc:511] Master 127.27.186.190:34969 requested a full tablet report, sending...
I20260809 15:02:31.362134 11937 ts_manager.cc:194] Registered new tserver with Master: d9a684f973ff4f60bba67d32698ec575 (127.27.186.130:38361)
W20260809 15:02:31.678479 12446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:31.678761 12446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:31.678921 12446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:31.685974 12446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:31.686101 12446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:31.686280 12446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:02:31.695034 12446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:31.696098 12446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:31.697721 12446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:31.709909 12456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:31.709946 12455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:31.710045 12460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:31.710836 12446 server_base.cc:1034] running on GCE node
I20260809 15:02:31.711306 12446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:31.712116 12446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:31.713291 12446 hybrid_clock.cc:648] HybridClock initialized: now 1786287751713244 us; error 45 us; skew 500 ppm
I20260809 15:02:31.713562 12446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:02:31.715646 12446 webserver.cc:460] Webserver started at http://127.27.186.131:36383/ using document root <none> and password file <none>
I20260809 15:02:31.716213 12446 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:31.716297 12446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:31.716547 12446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:31.719105 12446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "46dedb2fa1dd499eb60c4e222df5594d"
format_stamp: "Formatted at 2026-08-09 15:02:31 on dist-test-slave-q9h9"
server_key: "1a41fbac6f17e1b21846257a8fcef3bd"
server_key_iv: "5109714080edd4657e45a524bb992645"
server_key_version: "encryptionkey@0"
I20260809 15:02:31.719794 12446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "46dedb2fa1dd499eb60c4e222df5594d"
format_stamp: "Formatted at 2026-08-09 15:02:31 on dist-test-slave-q9h9"
server_key: "1a41fbac6f17e1b21846257a8fcef3bd"
server_key_iv: "5109714080edd4657e45a524bb992645"
server_key_version: "encryptionkey@0"
I20260809 15:02:31.724308 12446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 15:02:31.727363 12473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:31.728477 12446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:02:31.728596 12446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "46dedb2fa1dd499eb60c4e222df5594d"
format_stamp: "Formatted at 2026-08-09 15:02:31 on dist-test-slave-q9h9"
server_key: "1a41fbac6f17e1b21846257a8fcef3bd"
server_key_iv: "5109714080edd4657e45a524bb992645"
server_key_version: "encryptionkey@0"
I20260809 15:02:31.728758 12446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:31.734193 12446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:31.734884 12446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:31.735095 12446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:31.735867 12446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:31.737069 12446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:31.737150 12446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:31.737226 12446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:31.737278 12446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:31.767022 12446 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:44647
I20260809 15:02:31.767076 12642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:44647 every 8 connection(s)
I20260809 15:02:31.768121 12446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:02:31.774953 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:31.770269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:45621 (local address 127.27.186.190:34969)
0809 15:02:31.770443 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:31.770447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:31.771647 (+ 1200us) server_negotiation.cc:407] Connection header received
0809 15:02:31.773148 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:31.773152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:31.773204 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:31.773287 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:31.774388 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:31.774393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:31.774396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:31.774412 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:31.774445 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:31.774456 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:31.774556 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:31.774677 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 15:02:31.774726 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:02:31.775689 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:31.770724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:34969 (local address 127.27.186.131:45621)
0809 15:02:31.771532 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:31.771564 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:31.772938 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:31.773447 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:31.773454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:31.773533 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:31.774122 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:02:31.774124 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:31.774159 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:02:31.774236 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:31.774529 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:31.774534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:31.774539 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:02:31.774731 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 15:02:31.774912 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":247,"threads_started":1}
I20260809 15:02:31.776865 12643 heartbeater.cc:347] Connected to a master server at 127.27.186.190:34969
I20260809 15:02:31.777148 12643 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:31.777698 12643 heartbeater.cc:511] Master 127.27.186.190:34969 requested a full tablet report, sending...
I20260809 15:02:31.778406 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 12446
I20260809 15:02:31.778564 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:02:31.779068 28394 external_mini_cluster.cc:1428] Setting key 306bd186453dcb98326c0f50a5e4d997
I20260809 15:02:31.779116 11937 ts_manager.cc:194] Registered new tserver with Master: 46dedb2fa1dd499eb60c4e222df5594d (127.27.186.131:44647)
I20260809 15:02:31.781785 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 15:02:31.787292 12238 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:02:31.785926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.27.186.190:34969)
0809 15:02:31.786097 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:31.786102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:31.786617 (+ 515us) server_negotiation.cc:407] Connection header received
0809 15:02:31.786722 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:31.786725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:31.786768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:31.786836 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:31.787077 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46424: BlockingRecv error: recv got EOF from 127.0.0.1:46424 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:02:31.787894 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 12029
I20260809 15:02:31.790807 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:31.788743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.27.186.129:46469)
0809 15:02:31.789197 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:31.789203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:31.789226 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:02:31.789284 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:31.789288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:31.789444 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:31.789551 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:31.790293 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:31.790298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:31.790308 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:31.790319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:31.790355 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:31.790371 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:31.790472 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:31.790516 (+ 44us) server_negotiation.cc:299] Negotiation successful
0809 15:02:31.790590 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":202,"threads_started":1}
I20260809 15:02:31.791847 12225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:31.870267 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 12029
I20260809 15:02:31.895841 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 12240
I20260809 15:02:31.898969 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:31.896633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.27.186.130:38361)
0809 15:02:31.897126 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:31.897132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:31.897158 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:02:31.897218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:31.897223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:31.897422 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:31.897560 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:31.898363 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:31.898369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:31.898379 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:31.898398 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:31.898439 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:31.898453 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:31.898555 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:31.898615 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 15:02:31.898706 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260809 15:02:31.900138 12434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:31.977696 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 12240
I20260809 15:02:31.999495 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 12446
I20260809 15:02:32.003332 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:32.000445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.27.186.131:44647)
0809 15:02:32.001060 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:32.001067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:32.001096 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:02:32.001155 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:32.001161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:32.001364 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:32.001506 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:32.002618 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:32.002624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:32.002635 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:02:32.002651 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:32.002688 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:02:32.002702 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:32.002870 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:32.002951 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 15:02:32.003055 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":307,"threads_started":1}
I20260809 15:02:32.004495 12636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:32.083285 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 12446
I20260809 15:02:32.103875 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 11886
I20260809 15:02:32.105064 11986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:32.203423 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 11886
2026-08-09T15:02:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2261 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:02:32 dist-test-slave-q9h9 krb5kdc[12673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:02:32 dist-test-slave-q9h9 krb5kdc[12673](info): set up 2 sockets
Aug 09 15:02:32 dist-test-slave-q9h9 krb5kdc[12673](info): commencing operation
krb5kdc: starting...
W20260809 15:02:33.402701 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:02:33 dist-test-slave-q9h9 krb5kdc[12673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:02:33.616875 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43881
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:43881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:33.920673 12695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:33.920929 12695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:33.921054 12695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:33.927836 12695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:02:33.927938 12695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:33.928013 12695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:33.928085 12695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:02:33.928140 12695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:02:33.937100 12695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:43881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:33.938160 12695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:33.939285 12695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:33.950269 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:33.950224 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:33.950302 12708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:33.951236 12695 server_base.cc:1034] running on GCE node
I20260809 15:02:33.951896 12695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:33.953080 12695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:33.954267 12695 hybrid_clock.cc:648] HybridClock initialized: now 1786287753954218 us; error 36 us; skew 500 ppm
I20260809 15:02:33.954561 12695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:33 dist-test-slave-q9h9 krb5kdc[12673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287753, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:33.957355 12695 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:02:33.958660 12695 webserver.cc:460] Webserver started at http://127.27.186.190:41395/ using document root <none> and password file <none>
I20260809 15:02:33.959240 12695 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:33.959363 12695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:33.959635 12695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:33.962316 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "56573fc08ec74aed9b0b8346112f459c"
format_stamp: "Formatted at 2026-08-09 15:02:33 on dist-test-slave-q9h9"
server_key: "fec885152382d65bc107731ffd34e62a"
server_key_iv: "fc865723e8557524cdda24e3a3376894"
server_key_version: "encryptionkey@0"
I20260809 15:02:33.962994 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "56573fc08ec74aed9b0b8346112f459c"
format_stamp: "Formatted at 2026-08-09 15:02:33 on dist-test-slave-q9h9"
server_key: "fec885152382d65bc107731ffd34e62a"
server_key_iv: "fc865723e8557524cdda24e3a3376894"
server_key_version: "encryptionkey@0"
I20260809 15:02:33.967919 12695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 15:02:33.971024 12718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:33.972129 12695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:02:33.972282 12695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "56573fc08ec74aed9b0b8346112f459c"
format_stamp: "Formatted at 2026-08-09 15:02:33 on dist-test-slave-q9h9"
server_key: "fec885152382d65bc107731ffd34e62a"
server_key_iv: "fc865723e8557524cdda24e3a3376894"
server_key_version: "encryptionkey@0"
I20260809 15:02:33.972442 12695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:33.989704 12695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:33.993410 12695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:33.993644 12695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:34.010224 12695 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:43881
I20260809 15:02:34.010264 12801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:43881 every 8 connection(s)
I20260809 15:02:34.011399 12695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:02:34.014647 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:34.016286 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 12695
I20260809 15:02:34.016518 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:02:34.017324 28394 external_mini_cluster.cc:1428] Setting key d4e2af3f09a8fc71eb2d5935d71ecc00
I20260809 15:02:34.025520 12802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c: Bootstrap starting.
I20260809 15:02:34.028262 12802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:34.029173 12802 log.cc:824] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:02:34 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:34.031972 12802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c: No bootstrap required, opened a new log
I20260809 15:02:34.037927 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:34.019350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.27.186.190:43881)
0809 15:02:34.020081 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:34.020091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:34.020135 (+ 44us) server_negotiation.cc:407] Connection header received
0809 15:02:34.021764 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:34.021785 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:34.022169 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:34.022481 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:34.023951 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:34.025197 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:34.026509 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:34.027170 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:34.027325 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:34.031227 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:34.031234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:34.031246 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:34.031281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:34.034255 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:34.035058 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:34.035067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:34.035073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:34.035222 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:34.035590 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:34.035600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:34.035602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:34.036061 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:34.036309 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:34.036694 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 15:02:34.036998 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":332,"threads_started":1}
I20260809 15:02:34.040180 12802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56573fc08ec74aed9b0b8346112f459c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43881 } }
I20260809 15:02:34.040489 12802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:34.040582 12802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56573fc08ec74aed9b0b8346112f459c, State: Initialized, Role: FOLLOWER
I20260809 15:02:34.041123 12802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56573fc08ec74aed9b0b8346112f459c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43881 } }
I20260809 15:02:34.041327 12802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:02:34.041446 12802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:02:34.041586 12802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:34.043562 12802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56573fc08ec74aed9b0b8346112f459c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43881 } }
I20260809 15:02:34.043999 12802 leader_election.cc:302] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56573fc08ec74aed9b0b8346112f459c; no voters:
I20260809 15:02:34.044484 12802 leader_election.cc:288] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:02:34.044610 12813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:34.045228 12813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [term 1 LEADER]: Becoming Leader. State: Replica: 56573fc08ec74aed9b0b8346112f459c, State: Running, Role: LEADER
I20260809 15:02:34.045724 12813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56573fc08ec74aed9b0b8346112f459c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43881 } }
I20260809 15:02:34.046416 12802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:02:34.048589 12814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56573fc08ec74aed9b0b8346112f459c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56573fc08ec74aed9b0b8346112f459c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43881 } } }
I20260809 15:02:34.048638 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56573fc08ec74aed9b0b8346112f459c. Latest consensus state: current_term: 1 leader_uuid: "56573fc08ec74aed9b0b8346112f459c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56573fc08ec74aed9b0b8346112f459c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 43881 } } }
I20260809 15:02:34.048947 12814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:34.048995 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c [sys.catalog]: This master's current role is: LEADER
I20260809 15:02:34.051110 12820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:02:34.055764 12820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:02:34.064073 12820 catalog_manager.cc:1347] Generated new cluster ID: 6466a59567054da385a956fcff27a703
I20260809 15:02:34.064169 12820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:02:34.082410 12820 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:02:34.083465 12820 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:02:34.090451 12820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c: Generated new TSK 0
I20260809 15:02:34.091059 12820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:02:34.155478 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43881
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:34.474225 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:34.474468 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:34.474579 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:34.481679 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:34.481771 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:34.481920 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:02:34.490996 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:34.492062 12846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:34.493223 12846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:34.505182 12853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:34.505096 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:34.505131 12856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:34.506300 12846 server_base.cc:1034] running on GCE node
I20260809 15:02:34.506829 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:34.507645 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:34.508841 12846 hybrid_clock.cc:648] HybridClock initialized: now 1786287754508791 us; error 41 us; skew 500 ppm
I20260809 15:02:34.509110 12846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:34 dist-test-slave-q9h9 krb5kdc[12673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:34.515475 12846 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:02:34.516793 12846 webserver.cc:460] Webserver started at http://127.27.186.129:34781/ using document root <none> and password file <none>
I20260809 15:02:34.517370 12846 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:34.517470 12846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:34.517696 12846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:34.520309 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "aa20877602b942abad0ac706f2a10b01"
format_stamp: "Formatted at 2026-08-09 15:02:34 on dist-test-slave-q9h9"
server_key: "fc999289352dfe4a059a2b170b27f6c2"
server_key_iv: "f128d1eb1672ee32abfc0d982618348d"
server_key_version: "encryptionkey@0"
I20260809 15:02:34.520988 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "aa20877602b942abad0ac706f2a10b01"
format_stamp: "Formatted at 2026-08-09 15:02:34 on dist-test-slave-q9h9"
server_key: "fc999289352dfe4a059a2b170b27f6c2"
server_key_iv: "f128d1eb1672ee32abfc0d982618348d"
server_key_version: "encryptionkey@0"
I20260809 15:02:34.525624 12846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:02:34.528748 12872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:34.529891 12846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:02:34.530063 12846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "aa20877602b942abad0ac706f2a10b01"
format_stamp: "Formatted at 2026-08-09 15:02:34 on dist-test-slave-q9h9"
server_key: "fc999289352dfe4a059a2b170b27f6c2"
server_key_iv: "f128d1eb1672ee32abfc0d982618348d"
server_key_version: "encryptionkey@0"
I20260809 15:02:34.530236 12846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:34.547971 12846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:34.551589 12846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:34.551833 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:34.552650 12846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:34.553910 12846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:34.553987 12846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:34.554121 12846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:34.554174 12846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:34.584404 12846 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:41515
I20260809 15:02:34.584446 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:41515 every 8 connection(s)
I20260809 15:02:34.585466 12846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:02:34.586735 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 12846
I20260809 15:02:34.586970 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:02:34.587714 28394 external_mini_cluster.cc:1428] Setting key d6b3b8a31f07d4602fb0013d210ddce8
Aug 09 15:02:34 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:34.604607 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:34.587886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:55227 (local address 127.27.186.190:43881)
0809 15:02:34.588326 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:34.588331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:34.589747 (+ 1416us) server_negotiation.cc:407] Connection header received
0809 15:02:34.591392 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:34.591397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:34.591476 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:34.591591 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:34.593023 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:34.593944 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:34.595485 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:34.596019 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:34.596141 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:34.599240 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:34.599246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:34.599250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:34.599288 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:34.601795 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:34.602471 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:34.602476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:34.602480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:34.602630 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:34.603000 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:34.603004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:34.603007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:34.603332 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:34.603467 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:34.604060 (+ 593us) server_negotiation.cc:299] Negotiation successful
0809 15:02:34.604286 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":221,"threads_started":1}
I20260809 15:02:34.605250 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:34.588424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43881 (local address 127.27.186.129:55227)
0809 15:02:34.589217 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:34.589260 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:34.591130 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:34.591769 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:34.591776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:34.592341 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:34.592857 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:34.592874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:34.594136 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:34.594143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:34.595337 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:34.595349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:34.596196 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:34.596201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:34.596388 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:34.596944 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:34.596969 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:34.599038 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:34.601914 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:34.601920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:34.601930 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:34.602359 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:34.602728 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:34.602731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:34.602733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:34.602896 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:34.603440 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:34.603445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:34.603697 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 15:02:34.603909 (+ 212us) client_negotiation.cc:240] Negotiation successful
0809 15:02:34.604227 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":321,"threads_started":1}
I20260809 15:02:34.606696 13052 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43881
I20260809 15:02:34.606977 13052 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:34.607793 13052 heartbeater.cc:511] Master 127.27.186.190:43881 requested a full tablet report, sending...
I20260809 15:02:34.609762 12746 ts_manager.cc:194] Registered new tserver with Master: aa20877602b942abad0ac706f2a10b01 (127.27.186.129:41515)
I20260809 15:02:34.611145 12746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:55227
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:02:34.642449 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43881
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:34.951089 13061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:34.951335 13061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:34.951463 13061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:34.958926 13061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:34.959082 13061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:34.959252 13061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:02:34.971009 13061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:34.972132 13061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:34.973321 13061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:34.986186 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:34.986487 13070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:34.986675 13076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:34.987154 13061 server_base.cc:1034] running on GCE node
I20260809 15:02:34.987727 13061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:34.988551 13061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:34.989754 13061 hybrid_clock.cc:648] HybridClock initialized: now 1786287754989700 us; error 50 us; skew 500 ppm
I20260809 15:02:34.990065 13061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:34 dist-test-slave-q9h9 krb5kdc[12673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:34.993139 13061 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:02:34.994522 13061 webserver.cc:460] Webserver started at http://127.27.186.130:38017/ using document root <none> and password file <none>
I20260809 15:02:34.995137 13061 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:34.995252 13061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:34.995517 13061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:34.998191 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "04f604a17d944823b4881c59cb52baec"
format_stamp: "Formatted at 2026-08-09 15:02:34 on dist-test-slave-q9h9"
server_key: "50d3d4d175805da763befe782b89d340"
server_key_iv: "e0abe98b8ded810b35c8b141b4465f4b"
server_key_version: "encryptionkey@0"
I20260809 15:02:34.998903 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "04f604a17d944823b4881c59cb52baec"
format_stamp: "Formatted at 2026-08-09 15:02:34 on dist-test-slave-q9h9"
server_key: "50d3d4d175805da763befe782b89d340"
server_key_iv: "e0abe98b8ded810b35c8b141b4465f4b"
server_key_version: "encryptionkey@0"
I20260809 15:02:35.003691 13061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:02:35.007002 13084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:35.008306 13061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 15:02:35.008452 13061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "04f604a17d944823b4881c59cb52baec"
format_stamp: "Formatted at 2026-08-09 15:02:34 on dist-test-slave-q9h9"
server_key: "50d3d4d175805da763befe782b89d340"
server_key_iv: "e0abe98b8ded810b35c8b141b4465f4b"
server_key_version: "encryptionkey@0"
I20260809 15:02:35.008610 13061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:35.022693 13061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:35.027433 13061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:35.027722 13061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:35.028698 13061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:35.030177 13061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:35.030265 13061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:35.030373 13061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:35.030431 13061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:35.062575 13061 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:39309
I20260809 15:02:35.062618 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:39309 every 8 connection(s)
I20260809 15:02:35.063648 13061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:02:35.073810 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 13061
I20260809 15:02:35.073980 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:02:35.074417 28394 external_mini_cluster.cc:1428] Setting key 7af9fefb5faa778d4994d45201a3f96a
Aug 09 15:02:35 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:35.081547 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.065745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:35275 (local address 127.27.186.190:43881)
0809 15:02:35.065902 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.065906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.066969 (+ 1063us) server_negotiation.cc:407] Connection header received
0809 15:02:35.068670 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.068675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.068753 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.068840 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:35.070214 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.070932 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.072394 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.072791 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.072900 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.076140 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:35.076145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:35.076148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:35.076179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:35.078749 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.079432 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.079436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.079438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.079566 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.079969 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.079973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.079975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.080242 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:35.080374 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.081000 (+ 626us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.081224 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 15:02:35.082423 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.066182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43881 (local address 127.27.186.130:35275)
0809 15:02:35.066827 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.066864 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.068409 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.069064 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.069072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.069513 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:35.070039 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.070057 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.071078 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.071082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.072245 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.072262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.073001 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.073009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.073229 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.073807 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:35.073828 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:35.075947 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:35.078880 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:35.078889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:35.078903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:35.079311 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:35.079669 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:35.079674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:35.079678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:35.079853 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:35.080364 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:35.080371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:35.080657 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.080874 (+ 217us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.081333 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":254,"threads_started":1}
I20260809 15:02:35.083666 13256 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43881
I20260809 15:02:35.083930 13256 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:35.084553 13256 heartbeater.cc:511] Master 127.27.186.190:43881 requested a full tablet report, sending...
I20260809 15:02:35.085817 12746 ts_manager.cc:194] Registered new tserver with Master: 04f604a17d944823b4881c59cb52baec (127.27.186.130:39309)
I20260809 15:02:35.086632 12746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:35275
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:02:35.124449 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43881
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:02:35.437872 13262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:02:35.438146 13262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:02:35.438277 13262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:02:35.445047 13262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:02:35.445144 13262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:02:35.445300 13262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:02:35.454226 13262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:02:35.455313 13262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:02:35.456498 13262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:02:35.468312 13269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:35.468279 13273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:02:35.468348 13271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:02:35.469082 13262 server_base.cc:1034] running on GCE node
I20260809 15:02:35.469638 13262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:02:35.470490 13262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:02:35.471683 13262 hybrid_clock.cc:648] HybridClock initialized: now 1786287755471636 us; error 42 us; skew 500 ppm
I20260809 15:02:35.471952 13262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:02:35 dist-test-slave-q9h9 krb5kdc[12673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287755, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:02:35.474839 13262 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:02:35.476152 13262 webserver.cc:460] Webserver started at http://127.27.186.131:40253/ using document root <none> and password file <none>
I20260809 15:02:35.476756 13262 fs_manager.cc:359] Metadata directory not provided
I20260809 15:02:35.476873 13262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:02:35.477138 13262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:02:35.479789 13262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6"
format_stamp: "Formatted at 2026-08-09 15:02:35 on dist-test-slave-q9h9"
server_key: "ee51226af854cefc2a9de42bb04ec8a4"
server_key_iv: "e89111647cb241e63111058e85562dd6"
server_key_version: "encryptionkey@0"
I20260809 15:02:35.480507 13262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6"
format_stamp: "Formatted at 2026-08-09 15:02:35 on dist-test-slave-q9h9"
server_key: "ee51226af854cefc2a9de42bb04ec8a4"
server_key_iv: "e89111647cb241e63111058e85562dd6"
server_key_version: "encryptionkey@0"
I20260809 15:02:35.485204 13262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:02:35.488394 13282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:35.489545 13262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:02:35.489693 13262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6"
format_stamp: "Formatted at 2026-08-09 15:02:35 on dist-test-slave-q9h9"
server_key: "ee51226af854cefc2a9de42bb04ec8a4"
server_key_iv: "e89111647cb241e63111058e85562dd6"
server_key_version: "encryptionkey@0"
I20260809 15:02:35.489854 13262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:02:35.518846 13262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:02:35.522187 13262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:02:35.522471 13262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:02:35.523316 13262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:02:35.524493 13262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:02:35.524569 13262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:35.524679 13262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:02:35.524739 13262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:02:35.555092 13262 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:39505
I20260809 15:02:35.555159 13454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:39505 every 8 connection(s)
I20260809 15:02:35.556288 13262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:02:35.563169 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 13262
I20260809 15:02:35.563372 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:02:35.563890 28394 external_mini_cluster.cc:1428] Setting key c47b0840d27ee4d600b7ce019a64e28e
Aug 09 15:02:35 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287755, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:02:35.574074 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.558500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:51591 (local address 127.27.186.190:43881)
0809 15:02:35.558652 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.558656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.559706 (+ 1050us) server_negotiation.cc:407] Connection header received
0809 15:02:35.561367 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.561371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.561442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.561528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:35.562893 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.563966 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.565792 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.566234 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.566345 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.569450 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:35.569454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:35.569458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:35.569489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:35.571753 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.572351 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.572354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.572356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.572473 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.572798 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.572800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.572802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.573039 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:35.573144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.573646 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.573795 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:02:35.574889 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.558891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:43881 (local address 127.27.186.131:51591)
0809 15:02:35.559538 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.559584 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.561138 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.561700 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.561707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.562219 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:02:35.562731 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.562749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.564107 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.564111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.565501 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.565513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.566457 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.566464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.566680 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.567241 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:02:35.567264 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:02:35.569266 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:02:35.571883 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:35.571890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:35.571899 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:35.572269 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:35.572547 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:02:35.572550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:02:35.572552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:02:35.572719 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:02:35.573124 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:02:35.573129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:02:35.573385 (+ 256us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.573584 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.573917 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":261,"threads_started":1}
I20260809 15:02:35.576033 13456 heartbeater.cc:347] Connected to a master server at 127.27.186.190:43881
I20260809 15:02:35.576308 13456 heartbeater.cc:464] Registering TS with master...
I20260809 15:02:35.577019 13456 heartbeater.cc:511] Master 127.27.186.190:43881 requested a full tablet report, sending...
I20260809 15:02:35.578579 12744 ts_manager.cc:194] Registered new tserver with Master: bc2b422b5ffd4dd1b22c3512ac5b29b6 (127.27.186.131:39505)
I20260809 15:02:35.579370 12744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:51591
I20260809 15:02:35.590193 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:02:35.604977 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.594326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.27.186.190:43881)
0809 15:02:35.594471 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.594475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.594975 (+ 500us) server_negotiation.cc:407] Connection header received
0809 15:02:35.595096 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.595100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.595172 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.595240 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:35.596479 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.597160 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.598277 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.598644 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.598734 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.600655 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:35.600659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:35.600663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:35.600689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:35.602981 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.603584 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.603587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.603589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.603718 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.603966 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.603968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.603970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.604238 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:35.604408 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.604642 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.604780 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 15:02:35.608477 12744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 15:02:35.614245 13052 heartbeater.cc:503] Master 127.27.186.190:43881 was elected leader, sending a full tablet report...
W20260809 15:02:35.615258 12744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:02:35.671018 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.626233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39505 (local address 127.0.0.1:49380)
0809 15:02:35.661153 (+ 34920us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.661170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.661284 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.662363 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.662370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.662401 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:35.662783 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.662798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.664758 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.664764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.668234 (+ 3470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.668247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.670047 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.670054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.670283 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.670340 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.670461 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.670634 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34781,"thread_start_us":33702,"threads_started":1}
I20260809 15:02:35.675103 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.626972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.27.186.131:39505)
0809 15:02:35.653384 (+ 26412us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.653394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.661410 (+ 8016us) server_negotiation.cc:407] Connection header received
0809 15:02:35.661515 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.661522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.661798 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.661987 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:35.663006 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.664587 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.668414 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.669840 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.670047 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.670223 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.674556 (+ 4333us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.674728 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26224,"thread_start_us":26425,"threads_started":1}
I20260809 15:02:35.677505 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.625803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39309 (local address 127.0.0.1:42814)
0809 15:02:35.660321 (+ 34518us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.660367 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.660560 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.666944 (+ 6384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.666948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.666974 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:35.667699 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.667711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.672700 (+ 4989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.672705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.674771 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.674790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.676796 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.676803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.676974 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.677006 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.677083 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.677201 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34263,"threads_started":1}
I20260809 15:02:35.678098 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.626511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.27.186.130:39309)
0809 15:02:35.666408 (+ 39897us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.666416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.666442 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:02:35.666531 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.666539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.666750 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.666943 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:35.670311 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.672573 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.674974 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.676361 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.676547 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.677573 (+ 1026us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.677643 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.677807 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39731,"thread_start_us":40531,"threads_started":1}
I20260809 15:02:35.679296 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.627205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:41515 (local address 127.0.0.1:33484)
0809 15:02:35.671241 (+ 44036us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.671258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.671374 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.672368 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.672374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.672407 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:35.672596 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.672607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.674526 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.674533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.676595 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.676610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.678571 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.678576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.678746 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.678792 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.678887 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.679021 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43946,"thread_start_us":33666,"threads_started":1}
I20260809 15:02:35.679157 13362 tablet_service.cc:1467] Processing CreateTablet for tablet 5047461c3b8349be840efdfcc2697f38 (DEFAULT_TABLE table=test-table [id=3067aacefb0b459c875b396613a659cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:35.679991 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.626480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.27.186.129:41515)
0809 15:02:35.654058 (+ 27578us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.654066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.671358 (+ 17292us) server_negotiation.cc:407] Connection header received
0809 15:02:35.671504 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.671513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.671741 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.671948 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:35.672782 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.674373 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.676783 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.678154 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.678334 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.678481 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.679561 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.679689 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27381,"thread_start_us":27596,"threads_started":1}
I20260809 15:02:35.680814 13362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047461c3b8349be840efdfcc2697f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:35.682138 13157 tablet_service.cc:1467] Processing CreateTablet for tablet 5047461c3b8349be840efdfcc2697f38 (DEFAULT_TABLE table=test-table [id=3067aacefb0b459c875b396613a659cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:35.683596 13157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047461c3b8349be840efdfcc2697f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:35.683360 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 5047461c3b8349be840efdfcc2697f38 (DEFAULT_TABLE table=test-table [id=3067aacefb0b459c875b396613a659cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:02:35.684679 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047461c3b8349be840efdfcc2697f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:02:35.696959 13503 tablet_bootstrap.cc:492] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Bootstrap starting.
I20260809 15:02:35.697954 13502 tablet_bootstrap.cc:492] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Bootstrap starting.
I20260809 15:02:35.700497 13502 tablet_bootstrap.cc:654] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:35.700718 13503 tablet_bootstrap.cc:654] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:35.701375 13504 tablet_bootstrap.cc:492] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Bootstrap starting.
I20260809 15:02:35.701745 13502 log.cc:824] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:35.701946 13503 log.cc:824] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:35.704571 13504 tablet_bootstrap.cc:654] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Neither blocks nor log segments found. Creating new log.
I20260809 15:02:35.704726 13503 tablet_bootstrap.cc:492] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: No bootstrap required, opened a new log
I20260809 15:02:35.704723 13502 tablet_bootstrap.cc:492] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: No bootstrap required, opened a new log
I20260809 15:02:35.704994 13503 ts_tablet_manager.cc:1397] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 15:02:35.704994 13502 ts_tablet_manager.cc:1397] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260809 15:02:35.705668 13504 log.cc:824] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Log is configured to *not* fsync() on all Append() calls
I20260809 15:02:35.708091 13504 tablet_bootstrap.cc:492] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: No bootstrap required, opened a new log
I20260809 15:02:35.708316 13504 ts_tablet_manager.cc:1397] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260809 15:02:35.715037 13504 raft_consensus.cc:348] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.715363 13504 raft_consensus.cc:374] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:35.715142 13503 raft_consensus.cc:348] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.715142 13502 raft_consensus.cc:348] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.715451 13504 raft_consensus.cc:729] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f604a17d944823b4881c59cb52baec, State: Initialized, Role: FOLLOWER
I20260809 15:02:35.715543 13502 raft_consensus.cc:374] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:35.715543 13503 raft_consensus.cc:374] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:02:35.715644 13503 raft_consensus.cc:729] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa20877602b942abad0ac706f2a10b01, State: Initialized, Role: FOLLOWER
I20260809 15:02:35.715646 13502 raft_consensus.cc:729] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2b422b5ffd4dd1b22c3512ac5b29b6, State: Initialized, Role: FOLLOWER
I20260809 15:02:35.716212 13502 consensus_queue.cc:260] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.716234 13503 consensus_queue.cc:260] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.716238 13504 consensus_queue.cc:260] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.717509 13456 heartbeater.cc:503] Master 127.27.186.190:43881 was elected leader, sending a full tablet report...
I20260809 15:02:35.718175 13502 ts_tablet_manager.cc:1428] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260809 15:02:35.718205 13256 heartbeater.cc:503] Master 127.27.186.190:43881 was elected leader, sending a full tablet report...
I20260809 15:02:35.718497 13503 ts_tablet_manager.cc:1428] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260809 15:02:35.720481 13504 ts_tablet_manager.cc:1428] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260809 15:02:35.743355 13511 raft_consensus.cc:482] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:02:35.743551 13511 raft_consensus.cc:504] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.745154 13511 leader_election.cc:288] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04f604a17d944823b4881c59cb52baec (127.27.186.130:39309), aa20877602b942abad0ac706f2a10b01 (127.27.186.129:41515)
I20260809 15:02:35.750543 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.745593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:50483 (local address 127.27.186.129:41515)
0809 15:02:35.745829 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.745834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.745853 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:02:35.745961 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.745966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.746047 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.746126 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:35.746472 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.747208 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.748926 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.749710 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.749847 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.749917 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.750196 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.750326 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260809 15:02:35.750555 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.745393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:41515 (local address 127.27.186.131:50483)
0809 15:02:35.745665 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.745684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.745824 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.746162 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.746167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.746191 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:35.746352 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.746360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.747342 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.747348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.748784 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.748797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.749859 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.749864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.750050 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.750084 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.750163 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.750286 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":29}
I20260809 15:02:35.751251 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.745486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:42151 (local address 127.27.186.130:39309)
0809 15:02:35.745670 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.745676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.746250 (+ 574us) server_negotiation.cc:407] Connection header received
0809 15:02:35.746425 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.746429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.746491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.746561 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:02:35.747076 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.747835 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.749595 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.750399 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.750531 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.750596 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.750930 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.751054 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:02:35.751277 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.745495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39309 (local address 127.27.186.131:42151)
0809 15:02:35.746158 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 15:02:35.746172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:02:35.746310 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:02:35.746600 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:02:35.746605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:02:35.746628 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:02:35.746939 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.746948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.747964 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.747969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.749449 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:02:35.749461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.750589 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.750595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:02:35.750766 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.750795 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 15:02:35.750880 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 15:02:35.750968 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":1,"thread_start_us":388,"threads_started":1}
I20260809 15:02:35.751578 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047461c3b8349be840efdfcc2697f38" candidate_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa20877602b942abad0ac706f2a10b01" is_pre_election: true
I20260809 15:02:35.752090 12980 raft_consensus.cc:2454] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2b422b5ffd4dd1b22c3512ac5b29b6 in term 0.
I20260809 15:02:35.752213 13191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047461c3b8349be840efdfcc2697f38" candidate_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f604a17d944823b4881c59cb52baec" is_pre_election: true
I20260809 15:02:35.752640 13191 raft_consensus.cc:2454] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2b422b5ffd4dd1b22c3512ac5b29b6 in term 0.
I20260809 15:02:35.752653 13288 leader_election.cc:302] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa20877602b942abad0ac706f2a10b01, bc2b422b5ffd4dd1b22c3512ac5b29b6; no voters:
I20260809 15:02:35.752964 13511 raft_consensus.cc:2781] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:02:35.753085 13511 raft_consensus.cc:482] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:02:35.753186 13511 raft_consensus.cc:3037] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:35.755343 13511 raft_consensus.cc:504] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.755906 13511 leader_election.cc:288] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [CANDIDATE]: Term 1 election: Requested vote from peers 04f604a17d944823b4881c59cb52baec (127.27.186.130:39309), aa20877602b942abad0ac706f2a10b01 (127.27.186.129:41515)
I20260809 15:02:35.756175 13191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047461c3b8349be840efdfcc2697f38" candidate_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f604a17d944823b4881c59cb52baec"
I20260809 15:02:35.756364 13191 raft_consensus.cc:3037] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:35.756366 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047461c3b8349be840efdfcc2697f38" candidate_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa20877602b942abad0ac706f2a10b01"
I20260809 15:02:35.756546 12980 raft_consensus.cc:3037] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:02:35.759451 12980 raft_consensus.cc:2454] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2b422b5ffd4dd1b22c3512ac5b29b6 in term 1.
I20260809 15:02:35.759451 13191 raft_consensus.cc:2454] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2b422b5ffd4dd1b22c3512ac5b29b6 in term 1.
I20260809 15:02:35.759825 13293 leader_election.cc:302] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f604a17d944823b4881c59cb52baec, bc2b422b5ffd4dd1b22c3512ac5b29b6; no voters:
I20260809 15:02:35.760108 13511 raft_consensus.cc:2781] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:02:35.760478 13511 raft_consensus.cc:686] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 1 LEADER]: Becoming Leader. State: Replica: bc2b422b5ffd4dd1b22c3512ac5b29b6, State: Running, Role: LEADER
I20260809 15:02:35.760926 13511 consensus_queue.cc:237] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } }
I20260809 15:02:35.764781 12744 catalog_manager.cc:5495] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc2b422b5ffd4dd1b22c3512ac5b29b6 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2b422b5ffd4dd1b22c3512ac5b29b6" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39505 } health_report { overall_health: HEALTHY } } }
I20260809 15:02:35.798449 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.793333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.27.186.131:39505)
0809 15:02:35.793483 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.793488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.793715 (+ 227us) server_negotiation.cc:407] Connection header received
0809 15:02:35.793917 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.793920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.793997 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.794146 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:02:35.794528 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.795224 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.796507 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.796894 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.796997 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.797464 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:02:35.797885 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:02:35.798006 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.798146 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.798219 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 15:02:35.805254 13191 raft_consensus.cc:1261] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 1 FOLLOWER]: Refusing update from remote peer bc2b422b5ffd4dd1b22c3512ac5b29b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:35.805258 12980 raft_consensus.cc:1261] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 1 FOLLOWER]: Refusing update from remote peer bc2b422b5ffd4dd1b22c3512ac5b29b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:02:35.806069 13516 consensus_queue.cc:1035] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa20877602b942abad0ac706f2a10b01" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:02:35.806380 13511 consensus_queue.cc:1035] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f604a17d944823b4881c59cb52baec" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:02:35.807660 13461 tablet.cc:2369] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:02:35.815392 13257 tablet.cc:2369] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:02:35.834313 12743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:35.834645 12743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:02:35.838404 12743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56573fc08ec74aed9b0b8346112f459c: Sending DeleteTablet for 3 replicas of tablet 5047461c3b8349be840efdfcc2697f38
I20260809 15:02:35.839395 13157 tablet_service.cc:1514] Processing DeleteTablet for tablet 5047461c3b8349be840efdfcc2697f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:35 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:42814
I20260809 15:02:35.839459 12951 tablet_service.cc:1514] Processing DeleteTablet for tablet 5047461c3b8349be840efdfcc2697f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:35 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:33484
I20260809 15:02:35.840087 13362 tablet_service.cc:1514] Processing DeleteTablet for tablet 5047461c3b8349be840efdfcc2697f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:02:35 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:49380
I20260809 15:02:35.841109 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 12846
I20260809 15:02:35.853852 13539 tablet_replica.cc:330] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: stopping tablet replica
I20260809 15:02:35.855033 13539 raft_consensus.cc:2229] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:02:35.855641 13539 raft_consensus.cc:2258] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:35.856794 13538 tablet_replica.cc:330] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: stopping tablet replica
I20260809 15:02:35.857228 13538 raft_consensus.cc:2229] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:35.857717 13538 raft_consensus.cc:2258] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:35.858523 13537 tablet_replica.cc:330] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: stopping tablet replica
I20260809 15:02:35.858916 13537 raft_consensus.cc:2229] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:02:35.859380 13537 raft_consensus.cc:2258] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:02:35.859535 13538 ts_tablet_manager.cc:1905] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:35.861028 13539 ts_tablet_manager.cc:1905] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:35.861354 13537 ts_tablet_manager.cc:1905] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:02:35.866669 13538 ts_tablet_manager.cc:1918] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:35.866837 13538 log.cc:1196] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/5047461c3b8349be840efdfcc2697f38
I20260809 15:02:35.867363 13538 ts_tablet_manager.cc:1939] T 5047461c3b8349be840efdfcc2697f38 P aa20877602b942abad0ac706f2a10b01: Deleting consensus metadata
I20260809 15:02:35.867906 13539 ts_tablet_manager.cc:1918] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:35.867962 13537 ts_tablet_manager.cc:1918] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:02:35.868068 13539 log.cc:1196] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/5047461c3b8349be840efdfcc2697f38
I20260809 15:02:35.868218 13537 log.cc:1196] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/5047461c3b8349be840efdfcc2697f38
I20260809 15:02:35.868757 13537 ts_tablet_manager.cc:1939] T 5047461c3b8349be840efdfcc2697f38 P 04f604a17d944823b4881c59cb52baec: Deleting consensus metadata
I20260809 15:02:35.868981 12724 catalog_manager.cc:4841] TS aa20877602b942abad0ac706f2a10b01 (127.27.186.129:41515): tablet 5047461c3b8349be840efdfcc2697f38 (table test-table [id=3067aacefb0b459c875b396613a659cc]) successfully deleted
I20260809 15:02:35.870289 13539 ts_tablet_manager.cc:1939] T 5047461c3b8349be840efdfcc2697f38 P bc2b422b5ffd4dd1b22c3512ac5b29b6: Deleting consensus metadata
I20260809 15:02:35.870473 12730 catalog_manager.cc:4841] TS 04f604a17d944823b4881c59cb52baec (127.27.186.130:39309): tablet 5047461c3b8349be840efdfcc2697f38 (table test-table [id=3067aacefb0b459c875b396613a659cc]) successfully deleted
I20260809 15:02:35.871620 12726 catalog_manager.cc:4841] TS bc2b422b5ffd4dd1b22c3512ac5b29b6 (127.27.186.131:39505): tablet 5047461c3b8349be840efdfcc2697f38 (table test-table [id=3067aacefb0b459c875b396613a659cc]) successfully deleted
Aug 09 15:02:35 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:02:35.879284 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.842486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.27.186.129:41515)
0809 15:02:35.842640 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.842646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.852176 (+ 9530us) server_negotiation.cc:407] Connection header received
0809 15:02:35.852556 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.852562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.852636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.852700 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:35.856310 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.857444 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.868386 (+ 10942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.869026 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.869154 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.873844 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:35.873855 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:35.873867 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:35.873895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:35.876647 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.877342 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.877348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.877351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.877468 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:35.877830 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:35.877834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:35.877836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:35.878220 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:35.878500 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:35.878790 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 15:02:35.878992 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:02:35.879876 13045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:35.965384 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 12846
I20260809 15:02:35.989379 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 13061
Aug 09 15:02:35 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:02:36.005239 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:35.990222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.27.186.130:39309)
0809 15:02:35.990447 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:35.990454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:35.990478 (+ 24us) server_negotiation.cc:407] Connection header received
0809 15:02:35.990546 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:35.990552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:35.990634 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:35.990764 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:35.991858 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.993091 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.994507 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:35.995143 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:35.995271 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:35.999155 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:35.999173 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:35.999193 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:35.999233 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:36.002673 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:36.003299 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:36.003308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:36.003311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:36.003462 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:36.003763 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:36.003768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:36.003772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:36.004209 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:36.004516 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:36.004668 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 15:02:36.004888 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 15:02:36.005963 13249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:36.093545 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 13061
I20260809 15:02:36.127997 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 13262
Aug 09 15:02:36 dist-test-slave-q9h9 krb5kdc[12673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:02:36.143919 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:02:36.128753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.27.186.131:39505)
0809 15:02:36.128946 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 15:02:36.128952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:02:36.129021 (+ 69us) server_negotiation.cc:407] Connection header received
0809 15:02:36.129214 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:02:36.129220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:02:36.129295 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:02:36.129382 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:02:36.130720 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:36.131544 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:36.132748 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:02:36.133204 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:02:36.133315 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:02:36.137937 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:02:36.137950 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:02:36.137966 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:02:36.138005 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:02:36.140951 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:36.141762 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:36.141769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:36.141771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:36.141941 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:02:36.142397 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:02:36.142400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:02:36.142402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:02:36.142777 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:02:36.143023 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 15:02:36.143405 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 15:02:36.143566 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 15:02:36.144687 13450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:36.266223 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 13262
I20260809 15:02:36.291858 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 12695
I20260809 15:02:36.293058 12794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:02:36.398545 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 12695
2026-08-09T15:02:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4199 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:02:36 dist-test-slave-q9h9 krb5kdc[13549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:02:36 dist-test-slave-q9h9 krb5kdc[13549](info): set up 2 sockets
Aug 09 15:02:36 dist-test-slave-q9h9 krb5kdc[13549](info): commencing operation
krb5kdc: starting...
W20260809 15:02:37.611989 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:02:37 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:02:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/rangeradmin_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/rangeradmin_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/rangerlookup_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/rangerlookup_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
I20260809 15:02:37.907816 28394 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvIvrqI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/postgres -l logfile start
2026-08-09 15:02:39.989 UTC [13594] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 15:02:39.989 UTC [13594] LOG: listening on IPv4 address "127.27.186.148", port 59821
2026-08-09 15:02:39.991 UTC [13594] LOG: listening on Unix socket "/tmp/.s.PGSQL.59821"
2026-08-09 15:02:40.011 UTC [13598] LOG: database system was shut down at 2026-08-09 15:02:38 UTC
2026-08-09 15:02:40.014 UTC [13594] LOG: database system is ready to accept connections
I20260809 15:02:41.097955 28394 mini_postgres.cc:96] Postgres bound to 59821
2026-08-09 15:02:41.119 UTC [13611] FATAL: database "slave" does not exist
127.27.186.148:59821 - accepting connections
I20260809 15:02:41.120252 28394 mini_ranger.cc:161] Starting Ranger...
I20260809 15:02:41.145778 28394 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 15:02:41.519407 28394 mini_ranger.cc:87] Created ranger Postgres database
I20260809 15:02:41.519536 28394 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-admin
2026-08-09 15:02:41,814 [I] DB FLAVOR :POSTGRES
2026-08-09 15:02:41,815 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 15:02:41,815 [I] Checking connection..
2026-08-09 15:02:41,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 15:02:42,054 [I] Checking connection passed.
2026-08-09 15:02:42,054 [I] --------- Verifying version history table ---------
2026-08-09 15:02:42,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 15:02:42,304 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 15:02:42,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 15:02:42,532 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 15:02:42,533 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 15:02:42,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 15:02:42.756 UTC [13725] WARNING: there is no transaction in progress
2026-08-09 15:02:42,763 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 15:02:42,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 15:02:43,013 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 15:02:43,013 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 15:02:43,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 15:02:43,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 15:02:43,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9h9.c.gcp-upstream.internal','N') ;"
2026-08-09 15:02:43,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 15:02:44,007 [I] Table x_portal_user does not exist in database ranger
2026-08-09 15:02:44,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 15:02:44,253 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 15:02:44,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 15:02:44,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 15:02:44,740 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 15:02:44,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 15:02:45.606 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.620 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.635 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.648 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.661 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.679 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.739 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.745 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.757 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.765 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.774 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.786 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.795 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.800 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.805 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:45.810 UTC [13903] WARNING: there is no transaction in progress
2026-08-09 15:02:46,323 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 15:02:46,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 15:02:46,563 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 15:02:46,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 15:02:46,784 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 15:02:46,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 15:02:47,023 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 15:02:47,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 15:02:47,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 15:02:47,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9h9.c.gcp-upstream.internal';"
2026-08-09 15:02:47,737 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 15:02:47,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 15:02:47,984 [I] DB_PATCHES have already been applied
I20260809 15:02:47.990238 28394 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvIvrqI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 15:02:47.990329 28394 mini_ranger.cc:193] Using host: 127.27.186.148
I20260809 15:02:48.009315 28394 mini_ranger.cc:227] Ranger admin URL: http://127.27.186.148:33333
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 3:02:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 15:02:49.294602 28394 mini_ranger.cc:160] Time spent starting Ranger: real 8.174s user 0.002s sys 0.049s
Aug 09, 2026 3:03:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 3:03:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 3:03:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 3:03:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 3:03:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 15:03:10.506465 28394 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/rangerkms_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/rangerkms_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.186.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.186.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 15:03:10 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287790, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 15:03:10.660547 28394 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 15:03:10.680083 28394 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 15:03:11.348134 28394 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 15:03:11.348299 28394 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms
I20260809 15:03:11.348398 28394 mini_ranger_kms.cc:227] Using postgres at 127.27.186.148:59821
2026-08-09 15:03:11,808 [I] DB FLAVOR :POSTGRES
2026-08-09 15:03:11,808 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 15:03:11,808 [I] Checking connection
2026-08-09 15:03:11,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 15:03:12,037 [I] connection success
2026-08-09 15:03:12,037 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 15:03:12,037 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 15:03:12,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 15:03:12,259 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 15:03:12,259 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 15:03:12,259 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 15:03:12,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:59821/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 15:03:12,498 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 15:03:12.897982 28394 mini_ranger_kms.cc:310] Created kms service
I20260809 15:03:13.094205 28394 mini_ranger_kms.cc:326] Created kudu user
I20260809 15:03:13.135917 28394 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 15:03:13 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287790, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:03:13.513159 28394 mini_ranger_kms.cc:384] Added ranger policy
I20260809 15:03:13.514377 28394 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 15:03:13.514432 28394 mini_ranger_kms.cc:240] Using host: 127.27.186.148
I20260809 15:03:13.543058 28394 mini_ranger_kms.cc:276] Ranger KMS PID: 14187
I20260809 15:03:13.543200 28394 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.186.148:37051
Aug 09, 2026 3:03:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:03:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:03:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 3:03:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 3:03:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 3:03:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 15:03:15 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786287795, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.186.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:03:16.137320 28394 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.477s user 0.098s sys 0.391s
I20260809 15:03:16.137682 28394 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 15:03:16.137935 28394 mini_ranger_kms.cc:401] 127.27.186.148:37051/kms/v1/keys
Aug 09 15:03:16 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287795, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.186.148@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
Aug 09 15:03:16 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287795, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.186.148@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
Aug 09 15:03:18 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:03:18.860626 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38421
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:38421
--ranger_config_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:37051/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:03:19.187391 14270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:03:19.187654 14270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:03:19.187764 14270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:03:19.194979 14270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:03:19.195079 14270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:03:19.195132 14270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:03:19.195218 14270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:03:19.195276 14270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:03:19.205049 14270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.186.148:37051/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:38421
--ranger_config_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:03:19.206277 14270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:03:19.207561 14270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:03:19.220765 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:19.221299 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:19.221349 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:03:19.221830 14270 server_base.cc:1034] running on GCE node
I20260809 15:03:19.222554 14270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:03:19.223882 14270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:03:19.225081 14270 hybrid_clock.cc:648] HybridClock initialized: now 1786287799225033 us; error 48 us; skew 500 ppm
I20260809 15:03:19.225395 14270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:03:19 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287799, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:03:19.233934 14270 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:03:19.235541 14270 webserver.cc:460] Webserver started at http://127.27.186.190:33451/ using document root <none> and password file <none>
I20260809 15:03:19.236153 14270 fs_manager.cc:359] Metadata directory not provided
I20260809 15:03:19.236255 14270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:03:19.236544 14270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:03:19 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287799, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:03:19.285723 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "b8db843a7caf4df5bc0bced8f200ca40"
format_stamp: "Formatted at 2026-08-09 15:03:19 on dist-test-slave-q9h9"
server_key: "2e7fc146c9319e4f1abd45af43a2f0d7"
server_key_iv: "67f8ed3478f243e619dd8be9df9ce5e2"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:19.286650 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "b8db843a7caf4df5bc0bced8f200ca40"
format_stamp: "Formatted at 2026-08-09 15:03:19 on dist-test-slave-q9h9"
server_key: "2e7fc146c9319e4f1abd45af43a2f0d7"
server_key_iv: "67f8ed3478f243e619dd8be9df9ce5e2"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:19.291971 14270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 15:03:19.306725 14293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:19.308071 14270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:03:19.308235 14270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "b8db843a7caf4df5bc0bced8f200ca40"
format_stamp: "Formatted at 2026-08-09 15:03:19 on dist-test-slave-q9h9"
server_key: "2e7fc146c9319e4f1abd45af43a2f0d7"
server_key_iv: "67f8ed3478f243e619dd8be9df9ce5e2"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:19.308431 14270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:03:19.346649 14270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:03:19.350232 14270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:03:19.350497 14270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:03:19.367654 14270 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:38421
I20260809 15:03:19.367687 14372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:38421 every 8 connection(s)
I20260809 15:03:19.368842 14270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:03:19.372298 14373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:03:19.376382 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 14270
I20260809 15:03:19.376582 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:03:19.377364 28394 external_mini_cluster.cc:1428] Setting key 0455eb6ce31bb46530976f856988dafd
I20260809 15:03:19.383405 14373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40: Bootstrap starting.
I20260809 15:03:19.386849 14373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40: Neither blocks nor log segments found. Creating new log.
I20260809 15:03:19.387915 14373 log.cc:824] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40: Log is configured to *not* fsync() on all Append() calls
I20260809 15:03:19.391077 14373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40: No bootstrap required, opened a new log
Aug 09 15:03:19 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:03:19.398458 14373 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8db843a7caf4df5bc0bced8f200ca40" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38421 } }
I20260809 15:03:19.398777 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:19.379279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.27.186.190:38421)
0809 15:03:19.379975 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:19.379985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:19.380028 (+ 43us) server_negotiation.cc:407] Connection header received
0809 15:03:19.381969 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:19.381989 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:19.382367 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:19.382660 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:19.384200 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:19.385532 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:19.387298 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:19.387933 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:19.388089 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:19.392201 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:19.392208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:19.392222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:19.392257 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:19.395147 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:19.395760 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:19.395766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:19.395772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:19.395916 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:19.396264 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:19.396267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:19.396280 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:19.396779 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:19.397073 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:19.397519 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 15:03:19.397859 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":331,"threads_started":1}
I20260809 15:03:19.398794 14373 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:03:19.398933 14373 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8db843a7caf4df5bc0bced8f200ca40, State: Initialized, Role: FOLLOWER
I20260809 15:03:19.399482 14373 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8db843a7caf4df5bc0bced8f200ca40" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38421 } }
I20260809 15:03:19.399716 14373 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:03:19.399840 14373 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:03:19.400009 14373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:03:19.402679 14373 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8db843a7caf4df5bc0bced8f200ca40" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38421 } }
I20260809 15:03:19.403087 14373 leader_election.cc:302] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8db843a7caf4df5bc0bced8f200ca40; no voters:
I20260809 15:03:19.403563 14373 leader_election.cc:288] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:03:19.403680 14381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:03:19.404268 14381 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [term 1 LEADER]: Becoming Leader. State: Replica: b8db843a7caf4df5bc0bced8f200ca40, State: Running, Role: LEADER
I20260809 15:03:19.404809 14381 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8db843a7caf4df5bc0bced8f200ca40" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38421 } }
I20260809 15:03:19.405339 14373 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:03:19.407454 14383 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8db843a7caf4df5bc0bced8f200ca40. Latest consensus state: current_term: 1 leader_uuid: "b8db843a7caf4df5bc0bced8f200ca40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8db843a7caf4df5bc0bced8f200ca40" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38421 } } }
I20260809 15:03:19.407840 14383 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [sys.catalog]: This master's current role is: LEADER
I20260809 15:03:19.408349 14382 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8db843a7caf4df5bc0bced8f200ca40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8db843a7caf4df5bc0bced8f200ca40" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38421 } } }
I20260809 15:03:19.408587 14382 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40 [sys.catalog]: This master's current role is: LEADER
I20260809 15:03:19.411541 14393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:03:19.411960 14373 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 15:03:19.423897 14393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:03:19.434530 14393 catalog_manager.cc:1347] Generated new cluster ID: d2805a658bad4fafa608d84e2b87815c
I20260809 15:03:19.434638 14393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:03:19.475195 14393 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:03:19.476997 14393 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:03:19.487401 14393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40: Generated new TSK 0
I20260809 15:03:19.488325 14393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:03:20.215847 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:37051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38421
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:03:20 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786287800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 15:03:20.653378 14438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:03:20.653694 14438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:03:20.653895 14438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:03:20.665221 14438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:03:20.665385 14438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:03:20.665583 14438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:03:20.680495 14438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.186.148:37051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:03:20.681970 14438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:03:20.683641 14438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:03:20.706259 14455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:20.707049 14456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:20.708267 14460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:03:20.708974 14438 server_base.cc:1034] running on GCE node
I20260809 15:03:20.709661 14438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:03:20.710783 14438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:03:20.712004 14438 hybrid_clock.cc:648] HybridClock initialized: now 1786287800711962 us; error 29 us; skew 500 ppm
I20260809 15:03:20.712355 14438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:03:20 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:03:20.716445 14438 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:03:20.718372 14438 webserver.cc:460] Webserver started at http://127.27.186.129:39565/ using document root <none> and password file <none>
I20260809 15:03:20.719304 14438 fs_manager.cc:359] Metadata directory not provided
I20260809 15:03:20.719457 14438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:03:20.719832 14438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:03:20 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:03:20.746253 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "87c5ff6877964f1ea486b3335677362b"
format_stamp: "Formatted at 2026-08-09 15:03:20 on dist-test-slave-q9h9"
server_key: "e07d37a14a2b2d5a81c2eb830f505ee4"
server_key_iv: "05ef4aa63140cdf20d38ed45aace4f14"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:20.747324 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "87c5ff6877964f1ea486b3335677362b"
format_stamp: "Formatted at 2026-08-09 15:03:20 on dist-test-slave-q9h9"
server_key: "e07d37a14a2b2d5a81c2eb830f505ee4"
server_key_iv: "05ef4aa63140cdf20d38ed45aace4f14"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:20.763891 14438 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.011s
I20260809 15:03:20.787969 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:20.789629 14438 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260809 15:03:20.789870 14438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "87c5ff6877964f1ea486b3335677362b"
format_stamp: "Formatted at 2026-08-09 15:03:20 on dist-test-slave-q9h9"
server_key: "e07d37a14a2b2d5a81c2eb830f505ee4"
server_key_iv: "05ef4aa63140cdf20d38ed45aace4f14"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:20.790215 14438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:03:20.868134 14438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:03:20.871909 14438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:03:20.872200 14438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:03:20.873394 14438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:03:20.882427 14438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:03:20.882576 14438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:20.882789 14438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:03:20.882923 14438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:03:20 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:03:21.007993 14438 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:46873
I20260809 15:03:21.008023 14651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:46873 every 8 connection(s)
I20260809 15:03:21.009748 14438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:03:21.015549 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 14438
I20260809 15:03:21.015767 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:03:21.016588 28394 external_mini_cluster.cc:1428] Setting key ca571d8b60010770abe8c1a9257a74ce
Aug 09 15:03:21 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
I20260809 15:03:21.060221 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:21.012666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:34227 (local address 127.27.186.190:38421)
0809 15:03:21.038538 (+ 25872us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:21.038544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:21.038564 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:03:21.038622 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:21.038627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:21.038711 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:21.038817 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:21.044274 (+ 5457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.045397 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.047730 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.048368 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.048487 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:21.052942 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:21.052949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:21.052954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:21.053008 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:21.056493 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:21.057350 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:21.057356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:21.057360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:21.057510 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:21.058134 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:21.058140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:21.058143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:21.058525 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:21.058653 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:21.059600 (+ 947us) server_negotiation.cc:299] Negotiation successful
0809 15:03:21.059831 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25746,"thread_start_us":26250,"threads_started":1}
I20260809 15:03:21.062676 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:21.013515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:38421 (local address 127.27.186.129:34227)
0809 15:03:21.027123 (+ 13608us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:21.027180 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:21.029642 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:21.039350 (+ 9708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:21.039361 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:21.040018 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:03:21.040834 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:21.040858 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.045575 (+ 4717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.045581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:21.047569 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:21.047586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.048631 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.048641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:21.048916 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:21.049701 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:03:21.049764 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:03:21.052684 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:03:21.056653 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:03:21.056664 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:03:21.056681 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:03:21.057207 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:03:21.057650 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:03:21.057655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:03:21.057659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:03:21.057896 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:03:21.058750 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:03:21.058758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:03:21.059162 (+ 404us) client_negotiation.cc:769] Sending connection context
0809 15:03:21.060827 (+ 1665us) client_negotiation.cc:240] Negotiation successful
0809 15:03:21.061360 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13275,"thread_start_us":16394,"threads_started":1}
I20260809 15:03:21.064711 14652 heartbeater.cc:347] Connected to a master server at 127.27.186.190:38421
I20260809 15:03:21.065019 14652 heartbeater.cc:464] Registering TS with master...
I20260809 15:03:21.065814 14652 heartbeater.cc:511] Master 127.27.186.190:38421 requested a full tablet report, sending...
I20260809 15:03:21.068274 14321 ts_manager.cc:194] Registered new tserver with Master: 87c5ff6877964f1ea486b3335677362b (127.27.186.129:46873)
I20260809 15:03:21.070271 14321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:34227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:03:21.097312 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:37051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38421
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:03:21 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
W20260809 15:03:21.514863 14661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:03:21.515111 14661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:03:21.515233 14661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:03:21.522270 14661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:03:21.522377 14661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:03:21.522514 14661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:03:21.531581 14661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.186.148:37051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:03:21.532722 14661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:03:21.533905 14661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:03:21.545958 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:21.546056 14683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:21.546010 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:03:21.546864 14661 server_base.cc:1034] running on GCE node
I20260809 15:03:21.547417 14661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:03:21.548230 14661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:03:21.549423 14661 hybrid_clock.cc:648] HybridClock initialized: now 1786287801549379 us; error 38 us; skew 500 ppm
I20260809 15:03:21.549726 14661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:03:21 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287801, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:03:21.552691 14661 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:03:21.554050 14661 webserver.cc:460] Webserver started at http://127.27.186.130:44819/ using document root <none> and password file <none>
I20260809 15:03:21.554656 14661 fs_manager.cc:359] Metadata directory not provided
I20260809 15:03:21.554780 14661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:03:21.555070 14661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:03:21 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287801, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:03:21.570997 14661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "e28e3b03533842fca4199da609c844b1"
format_stamp: "Formatted at 2026-08-09 15:03:21 on dist-test-slave-q9h9"
server_key: "3f29aacdb1bded324af082b7554e12da"
server_key_iv: "53e3c6de7be3ff60d0056a9f811c246c"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:21.572062 14661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "e28e3b03533842fca4199da609c844b1"
format_stamp: "Formatted at 2026-08-09 15:03:21 on dist-test-slave-q9h9"
server_key: "3f29aacdb1bded324af082b7554e12da"
server_key_iv: "53e3c6de7be3ff60d0056a9f811c246c"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:21.578341 14661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 15:03:21.592634 14695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:21.593943 14661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:03:21.594130 14661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "e28e3b03533842fca4199da609c844b1"
format_stamp: "Formatted at 2026-08-09 15:03:21 on dist-test-slave-q9h9"
server_key: "3f29aacdb1bded324af082b7554e12da"
server_key_iv: "53e3c6de7be3ff60d0056a9f811c246c"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:21.594321 14661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:03:21.620847 14661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:03:21.624931 14661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:03:21.625217 14661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:03:21.626369 14661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:03:21.628140 14661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:03:21.628247 14661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:21.628371 14661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:03:21.628476 14661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 15:03:21.666373 14661 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:44547
I20260809 15:03:21.666443 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:44547 every 8 connection(s)
I20260809 15:03:21.667563 14661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:03:21.676980 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 14661
I20260809 15:03:21.677147 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:03:21.677641 28394 external_mini_cluster.cc:1428] Setting key 150380e79b97c71860daa89d7f6438f0
Aug 09 15:03:21 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287801, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:03:21.690174 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:21.670221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:35357 (local address 127.27.186.190:38421)
0809 15:03:21.670710 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:21.670716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:21.671708 (+ 992us) server_negotiation.cc:407] Connection header received
0809 15:03:21.674098 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:21.674102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:21.674168 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:21.674264 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:21.676056 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.676819 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.678604 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.679180 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.679315 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:21.683405 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:21.683412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:21.683417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:21.683457 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:21.686825 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:21.687653 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:21.687667 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:21.687670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:21.687824 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:21.688301 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:21.688306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:21.688310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:21.688647 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:21.688800 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:21.689514 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 15:03:21.689747 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260809 15:03:21.690924 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:21.670722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:38421 (local address 127.27.186.130:35357)
0809 15:03:21.671526 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:21.671579 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:21.673785 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:21.674498 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:21.674509 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:21.675116 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:03:21.675863 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:21.675888 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.676998 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.677002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:21.678450 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:21.678466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:21.679585 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:21.679595 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:21.679892 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:21.680610 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:03:21.680638 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:03:21.683176 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:03:21.686966 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:03:21.686974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:03:21.686988 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:03:21.687522 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:03:21.687938 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:03:21.687942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:03:21.687945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:03:21.688177 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:03:21.688784 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:03:21.688791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:03:21.689132 (+ 341us) client_negotiation.cc:769] Sending connection context
0809 15:03:21.689383 (+ 251us) client_negotiation.cc:240] Negotiation successful
0809 15:03:21.689822 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":308,"threads_started":1}
I20260809 15:03:21.692324 14860 heartbeater.cc:347] Connected to a master server at 127.27.186.190:38421
I20260809 15:03:21.692610 14860 heartbeater.cc:464] Registering TS with master...
I20260809 15:03:21.693372 14860 heartbeater.cc:511] Master 127.27.186.190:38421 requested a full tablet report, sending...
I20260809 15:03:21.695835 14321 ts_manager.cc:194] Registered new tserver with Master: e28e3b03533842fca4199da609c844b1 (127.27.186.130:44547)
I20260809 15:03:21.697276 14321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:35357
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:03:21.734699 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:37051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38421
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:03:22.045125 14869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:03:22.045354 14869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:03:22.045486 14869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:03:22.052359 14869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:03:22.052448 14869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:03:22.052558 14869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:03:22.061538 14869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.186.148:37051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:03:22.062657 14869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:03:22.063844 14869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 15:03:22.073604 14652 heartbeater.cc:503] Master 127.27.186.190:38421 was elected leader, sending a full tablet report...
W20260809 15:03:22.075899 14876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:22.075937 14880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:03:22.075851 14877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:03:22.076864 14869 server_base.cc:1034] running on GCE node
I20260809 15:03:22.077425 14869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:03:22.078277 14869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:03:22.079479 14869 hybrid_clock.cc:648] HybridClock initialized: now 1786287802079426 us; error 38 us; skew 500 ppm
I20260809 15:03:22.079758 14869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:03:22 dist-test-slave-q9h9 krb5kdc[13549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287802, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:03:22.082626 14869 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:03:22.083863 14869 webserver.cc:460] Webserver started at http://127.27.186.131:44131/ using document root <none> and password file <none>
I20260809 15:03:22.084450 14869 fs_manager.cc:359] Metadata directory not provided
I20260809 15:03:22.084565 14869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:03:22.084841 14869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:03:22 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287802, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:03:22.100643 14869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "50746d903db54549a977d487e7ad8539"
format_stamp: "Formatted at 2026-08-09 15:03:22 on dist-test-slave-q9h9"
server_key: "66b45df6b40081c40caa5e213e0ad216"
server_key_iv: "a17a8c305902670d81af7ab9a872eebc"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:22.101464 14869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "50746d903db54549a977d487e7ad8539"
format_stamp: "Formatted at 2026-08-09 15:03:22 on dist-test-slave-q9h9"
server_key: "66b45df6b40081c40caa5e213e0ad216"
server_key_iv: "a17a8c305902670d81af7ab9a872eebc"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:22.106489 14869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 15:03:22.122285 14888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:22.123610 14869 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260809 15:03:22.123749 14869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "50746d903db54549a977d487e7ad8539"
format_stamp: "Formatted at 2026-08-09 15:03:22 on dist-test-slave-q9h9"
server_key: "66b45df6b40081c40caa5e213e0ad216"
server_key_iv: "a17a8c305902670d81af7ab9a872eebc"
server_key_version: "kuduclusterkey@0"
I20260809 15:03:22.123946 14869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:03:22.152122 14869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:03:22.156105 14869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:03:22.156383 14869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:03:22.157310 14869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:03:22.158634 14869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:03:22.158707 14869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:22.158830 14869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:03:22.158885 14869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:03:22.189049 14869 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:44045
I20260809 15:03:22.189085 15063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:44045 every 8 connection(s)
I20260809 15:03:22.190223 14869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:03:22.193656 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 14869
I20260809 15:03:22.193854 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:03:22.194424 28394 external_mini_cluster.cc:1428] Setting key 4c9e77dc9e2aabee2680740b1420f83c
Aug 09 15:03:22 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287802, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:03:22.209226 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.192730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:46857 (local address 127.27.186.190:38421)
0809 15:03:22.193148 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.193152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.194095 (+ 943us) server_negotiation.cc:407] Connection header received
0809 15:03:22.196248 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.196253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.196337 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.196433 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:22.197883 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.198646 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.200147 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.200565 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.200676 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.203739 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:22.203744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:22.203747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:22.203787 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:22.206366 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.207010 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.207014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.207016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.207133 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.207558 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.207561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.207563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.207864 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:22.208014 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.208633 (+ 619us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.208844 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260809 15:03:22.210166 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.193138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:38421 (local address 127.27.186.131:46857)
0809 15:03:22.193921 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.193976 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.195954 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.196671 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.196679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.197091 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:03:22.197724 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.197743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.198797 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.198801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.200003 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.200016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.200793 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.200801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.201005 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.201577 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:03:22.201597 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:03:22.203529 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:03:22.206488 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:03:22.206497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:03:22.206511 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:03:22.206900 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:03:22.207234 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:03:22.207239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:03:22.207243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:03:22.207431 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:03:22.207991 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:03:22.207998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:03:22.208282 (+ 284us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.208525 (+ 243us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.209039 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":357,"threads_started":1}
I20260809 15:03:22.211573 15064 heartbeater.cc:347] Connected to a master server at 127.27.186.190:38421
I20260809 15:03:22.211869 15064 heartbeater.cc:464] Registering TS with master...
I20260809 15:03:22.212518 15064 heartbeater.cc:511] Master 127.27.186.190:38421 requested a full tablet report, sending...
I20260809 15:03:22.213871 14321 ts_manager.cc:194] Registered new tserver with Master: 50746d903db54549a977d487e7ad8539 (127.27.186.131:44045)
I20260809 15:03:22.214670 14321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:46857
I20260809 15:03:22.221231 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:03:22.236723 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.225435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.27.186.190:38421)
0809 15:03:22.225593 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.225598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.226063 (+ 465us) server_negotiation.cc:407] Connection header received
0809 15:03:22.226261 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.226265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.226331 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.226399 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:22.227784 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.228541 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.229829 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.230299 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.230396 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.232400 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:22.232405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:22.232408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:22.232434 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:22.234745 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.235288 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.235291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.235293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.235404 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.235665 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.235668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.235670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.235881 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:22.236051 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.236278 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.236440 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 15:03:22.240392 14321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:03:22.247999 14321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:03:22.269343 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.259603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.27.186.129:46873)
0809 15:03:22.261513 (+ 1910us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.261522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.261552 (+ 30us) server_negotiation.cc:407] Connection header received
0809 15:03:22.261665 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.261673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.261919 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.262748 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.262983 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.264663 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.266948 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.268203 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.268451 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.268627 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.268864 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.269029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":1550,"threads_started":1}
I20260809 15:03:22.269395 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.259316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:46873 (local address 127.0.0.1:56086)
0809 15:03:22.260006 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.260053 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.260215 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.262207 (+ 1992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.262216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.262252 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.262660 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.262674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.264831 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.264836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.266793 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.266806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.268451 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.268458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.268631 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.268679 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.268806 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.268980 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1,"thread_start_us":300,"threads_started":1}
I20260809 15:03:22.270574 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.260442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:44547 (local address 127.0.0.1:58900)
0809 15:03:22.262238 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.262257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.262540 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.263112 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.263116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.263141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.263539 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.263554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.265684 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.265689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.267840 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.267852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.269832 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.269838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.269978 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.270005 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.270109 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.270230 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"mutex_wait_us":4,"thread_start_us":1540,"threads_started":1}
I20260809 15:03:22.271183 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.259915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:44045 (local address 127.0.0.1:45208)
0809 15:03:22.260804 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.260821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.260930 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.263398 (+ 2468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.263403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.263433 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.263835 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.263846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.265362 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.265367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.267541 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.267555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.269312 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.269319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.269839 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.269873 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.270745 (+ 872us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.270858 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":20,"thread_start_us":901,"threads_started":1}
I20260809 15:03:22.271320 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.261044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.27.186.130:44547)
0809 15:03:22.261814 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.261826 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.262367 (+ 541us) server_negotiation.cc:407] Connection header received
0809 15:03:22.262765 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.262773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.262948 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.263126 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.263755 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.265547 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.268006 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.269378 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.269529 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.269680 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.270825 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.270980 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":702,"threads_started":1}
I20260809 15:03:22.271993 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.260564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.27.186.131:44045)
0809 15:03:22.262760 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.262767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.262803 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:03:22.262895 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.262902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.263141 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.263393 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.264022 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.265232 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.267728 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.269069 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.269923 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.270127 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.271498 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.271673 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2032,"thread_start_us":1750,"threads_started":1}
I20260809 15:03:22.273063 14556 tablet_service.cc:1467] Processing CreateTablet for tablet 79edeb8d415b4ba3ba4b04683e52778c (DEFAULT_TABLE table=test-table [id=35e75f045aad4ac4aa4735bbfed33ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:03:22.274266 14763 tablet_service.cc:1467] Processing CreateTablet for tablet 79edeb8d415b4ba3ba4b04683e52778c (DEFAULT_TABLE table=test-table [id=35e75f045aad4ac4aa4735bbfed33ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:03:22.274638 14556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79edeb8d415b4ba3ba4b04683e52778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:03:22.275302 14763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79edeb8d415b4ba3ba4b04683e52778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:03:22.275354 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 79edeb8d415b4ba3ba4b04683e52778c (DEFAULT_TABLE table=test-table [id=35e75f045aad4ac4aa4735bbfed33ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:03:22.276933 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79edeb8d415b4ba3ba4b04683e52778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:03:22.285871 15092 tablet_bootstrap.cc:492] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Bootstrap starting.
I20260809 15:03:22.288892 15094 tablet_bootstrap.cc:492] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Bootstrap starting.
I20260809 15:03:22.289237 15092 tablet_bootstrap.cc:654] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Neither blocks nor log segments found. Creating new log.
I20260809 15:03:22.290794 15092 log.cc:824] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Log is configured to *not* fsync() on all Append() calls
I20260809 15:03:22.291627 15095 tablet_bootstrap.cc:492] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Bootstrap starting.
I20260809 15:03:22.292491 15094 tablet_bootstrap.cc:654] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Neither blocks nor log segments found. Creating new log.
I20260809 15:03:22.293782 15094 log.cc:824] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Log is configured to *not* fsync() on all Append() calls
I20260809 15:03:22.294493 15095 tablet_bootstrap.cc:654] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Neither blocks nor log segments found. Creating new log.
I20260809 15:03:22.295787 15095 log.cc:824] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Log is configured to *not* fsync() on all Append() calls
I20260809 15:03:22.296432 15092 tablet_bootstrap.cc:492] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: No bootstrap required, opened a new log
I20260809 15:03:22.296712 15092 ts_tablet_manager.cc:1397] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260809 15:03:22.296780 15094 tablet_bootstrap.cc:492] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: No bootstrap required, opened a new log
I20260809 15:03:22.297008 15094 ts_tablet_manager.cc:1397] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 15:03:22.299364 15095 tablet_bootstrap.cc:492] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: No bootstrap required, opened a new log
I20260809 15:03:22.299611 15095 ts_tablet_manager.cc:1397] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:03:22.304208 15092 raft_consensus.cc:348] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.304512 15092 raft_consensus.cc:374] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:03:22.304613 15092 raft_consensus.cc:729] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28e3b03533842fca4199da609c844b1, State: Initialized, Role: FOLLOWER
I20260809 15:03:22.305176 15092 consensus_queue.cc:260] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.306216 14860 heartbeater.cc:503] Master 127.27.186.190:38421 was elected leader, sending a full tablet report...
I20260809 15:03:22.307129 15094 raft_consensus.cc:348] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.307588 15094 raft_consensus.cc:374] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:03:22.307674 15094 raft_consensus.cc:729] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c5ff6877964f1ea486b3335677362b, State: Initialized, Role: FOLLOWER
I20260809 15:03:22.307951 15092 ts_tablet_manager.cc:1428] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260809 15:03:22.308352 15094 consensus_queue.cc:260] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.309934 15095 raft_consensus.cc:348] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.310395 15095 raft_consensus.cc:374] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:03:22.310495 15095 raft_consensus.cc:729] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50746d903db54549a977d487e7ad8539, State: Initialized, Role: FOLLOWER
I20260809 15:03:22.311143 15095 consensus_queue.cc:260] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.312259 15064 heartbeater.cc:503] Master 127.27.186.190:38421 was elected leader, sending a full tablet report...
I20260809 15:03:22.313047 15095 ts_tablet_manager.cc:1428] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260809 15:03:22.314386 15094 ts_tablet_manager.cc:1428] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
W20260809 15:03:22.419445 14862 tablet.cc:2369] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:03:22.441610 15065 tablet.cc:2369] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:03:22.545928 14653 tablet.cc:2369] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:03:22.549589 15101 raft_consensus.cc:482] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:03:22.549846 15101 raft_consensus.cc:504] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.551512 15101 leader_election.cc:288] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c5ff6877964f1ea486b3335677362b (127.27.186.129:46873), 50746d903db54549a977d487e7ad8539 (127.27.186.131:44045)
I20260809 15:03:22.556600 15104 raft_consensus.cc:482] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:03:22.556941 15104 raft_consensus.cc:504] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.558073 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.551786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:45357 (local address 127.27.186.129:46873)
0809 15:03:22.551994 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.552001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.552813 (+ 812us) server_negotiation.cc:407] Connection header received
0809 15:03:22.552970 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.552974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.553033 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.553105 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.553673 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.554407 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.556138 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.557037 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.557177 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.557261 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.557698 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.557821 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:03:22.558117 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.552024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:40041 (local address 127.27.186.131:44045)
0809 15:03:22.552219 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.552226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.552621 (+ 395us) server_negotiation.cc:407] Connection header received
0809 15:03:22.552805 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.552809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.552871 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.552948 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.553512 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.554291 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.556081 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.557143 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.557312 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.557380 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.557730 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.557849 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:03:22.558373 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.551867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:46873 (local address 127.27.186.130:45357)
0809 15:03:22.552698 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.552718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.552842 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.553143 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.553148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.553176 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.553528 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.553538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.554543 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.554547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.555992 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.556004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.557234 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.557240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.557518 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.557553 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.557651 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.557780 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":538,"threads_started":1}
I20260809 15:03:22.558763 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.551728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:44045 (local address 127.27.186.130:40041)
0809 15:03:22.552512 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.552530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.552671 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.552979 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.552983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.553009 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.553370 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.553382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.554411 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.554416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.555936 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.555948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.557420 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.557426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.557569 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.557594 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.558254 (+ 660us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.558390 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":1,"thread_start_us":460,"threads_started":1}
I20260809 15:03:22.559661 15104 leader_election.cc:288] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c5ff6877964f1ea486b3335677362b (127.27.186.129:46873), e28e3b03533842fca4199da609c844b1 (127.27.186.130:44547)
I20260809 15:03:22.559542 14590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79edeb8d415b4ba3ba4b04683e52778c" candidate_uuid: "e28e3b03533842fca4199da609c844b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5ff6877964f1ea486b3335677362b" is_pre_election: true
I20260809 15:03:22.560377 14590 raft_consensus.cc:2454] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e28e3b03533842fca4199da609c844b1 in term 0.
I20260809 15:03:22.561347 14696 leader_election.cc:302] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c5ff6877964f1ea486b3335677362b, e28e3b03533842fca4199da609c844b1; no voters:
I20260809 15:03:22.561282 14999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79edeb8d415b4ba3ba4b04683e52778c" candidate_uuid: "e28e3b03533842fca4199da609c844b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50746d903db54549a977d487e7ad8539" is_pre_election: true
I20260809 15:03:22.561756 15101 raft_consensus.cc:2781] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:03:22.561841 14999 raft_consensus.cc:2454] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e28e3b03533842fca4199da609c844b1 in term 0.
I20260809 15:03:22.561923 15101 raft_consensus.cc:482] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:03:22.562059 15101 raft_consensus.cc:3037] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:03:22.564641 15101 raft_consensus.cc:504] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.565276 15101 leader_election.cc:288] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [CANDIDATE]: Term 1 election: Requested vote from peers 87c5ff6877964f1ea486b3335677362b (127.27.186.129:46873), 50746d903db54549a977d487e7ad8539 (127.27.186.131:44045)
I20260809 15:03:22.565694 14999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79edeb8d415b4ba3ba4b04683e52778c" candidate_uuid: "e28e3b03533842fca4199da609c844b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50746d903db54549a977d487e7ad8539"
I20260809 15:03:22.565866 14999 raft_consensus.cc:3037] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:03:22.566078 14590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79edeb8d415b4ba3ba4b04683e52778c" candidate_uuid: "e28e3b03533842fca4199da609c844b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5ff6877964f1ea486b3335677362b"
I20260809 15:03:22.566270 14590 raft_consensus.cc:3037] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:03:22.566977 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.559174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:46873 (local address 127.27.186.131:48175)
0809 15:03:22.559442 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.559458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.559551 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.560113 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.560118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.560155 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.560331 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.560342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.562407 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.562411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.564615 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.564634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.566216 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.566221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.566457 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.566492 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.566592 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.566728 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260809 15:03:22.567443 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.559996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:35079 (local address 127.27.186.130:44547)
0809 15:03:22.560213 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.560219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.560552 (+ 333us) server_negotiation.cc:407] Connection header received
0809 15:03:22.560779 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.560784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.560850 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.560908 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.561892 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.563008 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.565217 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.566354 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.566506 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.566572 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.567110 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.567231 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:03:22.567620 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.559391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:48175 (local address 127.27.186.129:46873)
0809 15:03:22.559557 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.559562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.559581 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:03:22.559632 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.559638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.559708 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.559816 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:03:22.560515 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.562237 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.564775 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.565877 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.566061 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.566126 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.567153 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.567292 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:03:22.568040 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.559857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:44547 (local address 127.27.186.131:35079)
0809 15:03:22.560470 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0809 15:03:22.560486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:03:22.560635 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:03:22.561086 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:03:22.561090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:03:22.561116 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:03:22.561686 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.561700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.563146 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.563152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.565079 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:03:22.565093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.566726 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.566746 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:03:22.566912 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.566949 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 15:03:22.567589 (+ 640us) client_negotiation.cc:240] Negotiation successful
0809 15:03:22.567742 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":575,"threads_started":1}
I20260809 15:03:22.569083 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79edeb8d415b4ba3ba4b04683e52778c" candidate_uuid: "50746d903db54549a977d487e7ad8539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28e3b03533842fca4199da609c844b1" is_pre_election: true
I20260809 15:03:22.568804 14589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79edeb8d415b4ba3ba4b04683e52778c" candidate_uuid: "50746d903db54549a977d487e7ad8539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5ff6877964f1ea486b3335677362b" is_pre_election: true
I20260809 15:03:22.569350 14999 raft_consensus.cc:2454] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e28e3b03533842fca4199da609c844b1 in term 1.
I20260809 15:03:22.569495 14792 raft_consensus.cc:2379] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50746d903db54549a977d487e7ad8539 in current term 1: Already voted for candidate e28e3b03533842fca4199da609c844b1 in this term.
I20260809 15:03:22.569957 14699 leader_election.cc:302] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50746d903db54549a977d487e7ad8539, e28e3b03533842fca4199da609c844b1; no voters:
I20260809 15:03:22.570308 15101 raft_consensus.cc:2781] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:03:22.570417 14889 leader_election.cc:302] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50746d903db54549a977d487e7ad8539; no voters: 87c5ff6877964f1ea486b3335677362b, e28e3b03533842fca4199da609c844b1
I20260809 15:03:22.570773 15104 raft_consensus.cc:2726] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:03:22.571175 14590 raft_consensus.cc:2454] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e28e3b03533842fca4199da609c844b1 in term 1.
I20260809 15:03:22.572108 15101 raft_consensus.cc:686] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 1 LEADER]: Becoming Leader. State: Replica: e28e3b03533842fca4199da609c844b1, State: Running, Role: LEADER
I20260809 15:03:22.572610 15101 consensus_queue.cc:237] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } }
I20260809 15:03:22.577451 14320 catalog_manager.cc:5495] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e28e3b03533842fca4199da609c844b1 (127.27.186.130). New cstate: current_term: 1 leader_uuid: "e28e3b03533842fca4199da609c844b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e28e3b03533842fca4199da609c844b1" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 } health_report { overall_health: UNKNOWN } } }
I20260809 15:03:22.630170 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.625240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.27.186.130:44547)
0809 15:03:22.625421 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.625427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.625505 (+ 78us) server_negotiation.cc:407] Connection header received
0809 15:03:22.625686 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.625689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.625768 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.625886 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:03:22.626229 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.626949 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.628345 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.628730 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.628820 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.629229 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:03:22.629662 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:03:22.629762 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.629839 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.629897 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:03:22.637274 14999 raft_consensus.cc:1261] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 1 FOLLOWER]: Refusing update from remote peer e28e3b03533842fca4199da609c844b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:03:22.637274 14590 raft_consensus.cc:1261] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 1 FOLLOWER]: Refusing update from remote peer e28e3b03533842fca4199da609c844b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:03:22.637993 15109 consensus_queue.cc:1035] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c5ff6877964f1ea486b3335677362b" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:03:22.638362 15101 consensus_queue.cc:1035] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50746d903db54549a977d487e7ad8539" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:03:22.661065 14321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:03:22.661295 14321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:03:22.664561 14321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8db843a7caf4df5bc0bced8f200ca40: Sending DeleteTablet for 3 replicas of tablet 79edeb8d415b4ba3ba4b04683e52778c
I20260809 15:03:22.665719 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet 79edeb8d415b4ba3ba4b04683e52778c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:03:22 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45208
I20260809 15:03:22.665717 14763 tablet_service.cc:1514] Processing DeleteTablet for tablet 79edeb8d415b4ba3ba4b04683e52778c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:03:22 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:58900
I20260809 15:03:22.665735 14556 tablet_service.cc:1514] Processing DeleteTablet for tablet 79edeb8d415b4ba3ba4b04683e52778c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:03:22 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:56086
I20260809 15:03:22.672852 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 14438
I20260809 15:03:22.681485 15125 tablet_replica.cc:330] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: stopping tablet replica
I20260809 15:03:22.681867 15125 raft_consensus.cc:2229] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:03:22.682327 15125 raft_consensus.cc:2258] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:03:22.683027 15126 tablet_replica.cc:330] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: stopping tablet replica
I20260809 15:03:22.683375 15126 raft_consensus.cc:2229] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:03:22.683812 15126 raft_consensus.cc:2258] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:03:22.685663 15125 ts_tablet_manager.cc:1905] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:03:22.686322 15126 ts_tablet_manager.cc:1905] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:03:22.692831 15125 ts_tablet_manager.cc:1918] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:03:22.692991 15125 log.cc:1196] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/79edeb8d415b4ba3ba4b04683e52778c
I20260809 15:03:22.693143 15126 ts_tablet_manager.cc:1918] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:03:22.693385 15126 log.cc:1196] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/79edeb8d415b4ba3ba4b04683e52778c
I20260809 15:03:22.693503 15125 ts_tablet_manager.cc:1939] T 79edeb8d415b4ba3ba4b04683e52778c P 87c5ff6877964f1ea486b3335677362b: Deleting consensus metadata
I20260809 15:03:22.693969 15126 ts_tablet_manager.cc:1939] T 79edeb8d415b4ba3ba4b04683e52778c P 50746d903db54549a977d487e7ad8539: Deleting consensus metadata
I20260809 15:03:22.695312 14297 catalog_manager.cc:4841] TS 87c5ff6877964f1ea486b3335677362b (127.27.186.129:46873): tablet 79edeb8d415b4ba3ba4b04683e52778c (table test-table [id=35e75f045aad4ac4aa4735bbfed33ce3]) successfully deleted
I20260809 15:03:22.695648 14302 catalog_manager.cc:4841] TS 50746d903db54549a977d487e7ad8539 (127.27.186.131:44045): tablet 79edeb8d415b4ba3ba4b04683e52778c (table test-table [id=35e75f045aad4ac4aa4735bbfed33ce3]) successfully deleted
I20260809 15:03:22.703111 15127 tablet_replica.cc:330] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: stopping tablet replica
I20260809 15:03:22.703533 15127 raft_consensus.cc:2229] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:03:22.704159 15127 raft_consensus.cc:2258] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:03:22.705932 15127 ts_tablet_manager.cc:1905] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 15:03:22 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:03:22.713564 15127 ts_tablet_manager.cc:1918] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:03:22.713814 15127 log.cc:1196] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/79edeb8d415b4ba3ba4b04683e52778c
I20260809 15:03:22.714581 15127 ts_tablet_manager.cc:1939] T 79edeb8d415b4ba3ba4b04683e52778c P e28e3b03533842fca4199da609c844b1: Deleting consensus metadata
I20260809 15:03:22.716799 14297 catalog_manager.cc:4841] TS e28e3b03533842fca4199da609c844b1 (127.27.186.130:44547): tablet 79edeb8d415b4ba3ba4b04683e52778c (table test-table [id=35e75f045aad4ac4aa4735bbfed33ce3]) successfully deleted
I20260809 15:03:22.722838 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.674056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.27.186.129:46873)
0809 15:03:22.674242 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.674249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.693290 (+ 19041us) server_negotiation.cc:407] Connection header received
0809 15:03:22.694206 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.694212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.694315 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.694392 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:22.702715 (+ 8323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.703893 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.706961 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.707575 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.707695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.714858 (+ 7163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:22.714875 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:22.714891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:22.714931 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:22.719482 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.720503 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.720509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.720511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.720696 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.721205 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.721210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.721212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.721663 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:22.721973 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.722288 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.722471 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 15:03:22.723865 14645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:03:22.813714 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 14438
I20260809 15:03:22.842154 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 14661
Aug 09 15:03:22 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:03:22.859319 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.843050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.27.186.130:44547)
0809 15:03:22.843274 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.843280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.843332 (+ 52us) server_negotiation.cc:407] Connection header received
0809 15:03:22.843399 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.843404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.843483 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.843593 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:22.844812 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.845970 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.847451 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.848133 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.848267 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.852423 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:22.852441 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:22.852464 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:22.852519 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:22.856471 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.857085 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.857095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.857099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.857282 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.857619 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.857624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.857628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.858110 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:22.858430 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.858739 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.858962 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:03:22.860009 14851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:03:22.966719 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 14661
I20260809 15:03:22.986624 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 14869
Aug 09 15:03:22 dist-test-slave-q9h9 krb5kdc[13549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:03:23.000268 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:03:22.987287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.27.186.131:44045)
0809 15:03:22.987469 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 15:03:22.987473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:03:22.987491 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:03:22.987636 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:03:22.987639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:03:22.987700 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:03:22.987801 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:03:22.988994 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.989727 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.991045 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:03:22.991457 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:03:22.991550 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:03:22.995402 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:03:22.995415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:03:22.995431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:03:22.995461 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:03:22.997946 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.998560 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.998568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.998570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.998712 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:03:22.998992 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:03:22.998994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:03:22.998996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:03:22.999333 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:03:22.999553 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 15:03:22.999786 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 15:03:22.999945 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:03:23.000837 15058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:03:23.088579 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 14869
I20260809 15:03:23.111016 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 14270
I20260809 15:03:23.112133 14368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:03:23.219915 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 14270
I20260809 15:03:23.249466 28394 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 15:03:23.593807 28394 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 15:03:23.594043 28394 mini_ranger.cc:66] Stopping Ranger...
I20260809 15:03:23.937732 28394 mini_ranger.cc:68] Stopped Ranger
2026-08-09 15:03:23.937 UTC [13594] LOG: received smart shutdown request
2026-08-09 15:03:23.943 UTC [13594] LOG: background worker "logical replication launcher" (PID 13607) exited with exit code 1
2026-08-09 15:03:23.943 UTC [13599] LOG: shutting down
2026-08-09 15:03:24.068 UTC [13594] LOG: database system is shut down
2026-08-09T15:03:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47774 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:03:24 dist-test-slave-q9h9 krb5kdc[15151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:03:24 dist-test-slave-q9h9 krb5kdc[15151](info): set up 2 sockets
Aug 09 15:03:24 dist-test-slave-q9h9 krb5kdc[15151](info): commencing operation
krb5kdc: starting...
W20260809 15:03:25.388146 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:03:25 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:03:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/rangeradmin_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/rangeradmin_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/rangerlookup_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/rangerlookup_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
I20260809 15:03:25.681679 28394 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvIvrqI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/postgres -l logfile start
2026-08-09 15:03:27.718 UTC [15203] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 15:03:27.718 UTC [15203] LOG: listening on IPv4 address "127.27.186.148", port 58073
2026-08-09 15:03:27.720 UTC [15203] LOG: listening on Unix socket "/tmp/.s.PGSQL.58073"
2026-08-09 15:03:27.741 UTC [15209] LOG: database system was shut down at 2026-08-09 15:03:26 UTC
2026-08-09 15:03:27.744 UTC [15203] LOG: database system is ready to accept connections
I20260809 15:03:28.828049 28394 mini_postgres.cc:96] Postgres bound to 58073
2026-08-09 15:03:28.849 UTC [15218] FATAL: database "slave" does not exist
127.27.186.148:58073 - accepting connections
I20260809 15:03:28.849741 28394 mini_ranger.cc:161] Starting Ranger...
I20260809 15:03:28.874504 28394 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 15:03:29.239140 28394 mini_ranger.cc:87] Created ranger Postgres database
I20260809 15:03:29.239274 28394 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-admin
2026-08-09 15:03:29,518 [I] DB FLAVOR :POSTGRES
2026-08-09 15:03:29,518 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 15:03:29,518 [I] Checking connection..
2026-08-09 15:03:29,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 15:03:29,761 [I] Checking connection passed.
2026-08-09 15:03:29,761 [I] --------- Verifying version history table ---------
2026-08-09 15:03:29,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 15:03:30,013 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 15:03:30,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 15:03:30,265 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 15:03:30,265 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 15:03:30,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 15:03:30.508 UTC [15317] WARNING: there is no transaction in progress
2026-08-09 15:03:30,516 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 15:03:30,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 15:03:30,756 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 15:03:30,757 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 15:03:30,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 15:03:30,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 15:03:31,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9h9.c.gcp-upstream.internal','N') ;"
2026-08-09 15:03:31,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 15:03:31,699 [I] Table x_portal_user does not exist in database ranger
2026-08-09 15:03:31,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 15:03:31,946 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 15:03:31,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 15:03:32,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 15:03:32,419 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 15:03:32,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 15:03:33.272 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.286 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.299 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.312 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.325 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.343 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.407 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.413 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.426 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.434 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.443 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.455 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.464 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.472 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.479 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:33.487 UTC [15503] WARNING: there is no transaction in progress
2026-08-09 15:03:34,022 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 15:03:34,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 15:03:34,268 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 15:03:34,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 15:03:34,504 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 15:03:34,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 15:03:34,752 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 15:03:34,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 15:03:34,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 15:03:35,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9h9.c.gcp-upstream.internal';"
2026-08-09 15:03:35,463 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 15:03:35,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 15:03:35,685 [I] DB_PATCHES have already been applied
I20260809 15:03:35.691094 28394 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvIvrqI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 15:03:35.691190 28394 mini_ranger.cc:193] Using host: 127.27.186.148
I20260809 15:03:35.709805 28394 mini_ranger.cc:227] Ranger admin URL: http://127.27.186.148:57085
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 3:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 15:03:37.014284 28394 mini_ranger.cc:160] Time spent starting Ranger: real 8.165s user 0.003s sys 0.046s
Aug 09, 2026 3:03:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 3:03:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 3:03:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 3:03:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 3:03:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 15:03:58.056358 28394 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.186.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.186.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/rangerkms_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.186.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/rangerkms_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.186.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.186.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.186.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/HTTP_127.27.186.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 15:03:58 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287838, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 15:03:58.216459 28394 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 15:03:58.235921 28394 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 15:03:58.897840 28394 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 15:03:58.897944 28394 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms
I20260809 15:03:58.897985 28394 mini_ranger_kms.cc:227] Using postgres at 127.27.186.148:58073
2026-08-09 15:03:59,248 [I] DB FLAVOR :POSTGRES
2026-08-09 15:03:59,248 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 15:03:59,248 [I] Checking connection
2026-08-09 15:03:59,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 15:03:59,499 [I] connection success
2026-08-09 15:03:59,499 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 15:03:59,499 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 15:03:59,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 15:03:59,723 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 15:03:59,723 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 15:03:59,723 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 15:03:59,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.186.148:58073/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 15:03:59,973 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 15:04:00.400589 28394 mini_ranger_kms.cc:310] Created kms service
I20260809 15:04:00.598989 28394 mini_ranger_kms.cc:326] Created kudu user
I20260809 15:04:00.639741 28394 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 15:04:00 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287838, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:01.000512 28394 mini_ranger_kms.cc:384] Added ranger policy
I20260809 15:04:01.000824 28394 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvIvrqI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvIvrqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 15:04:01.000876 28394 mini_ranger_kms.cc:240] Using host: 127.27.186.148
I20260809 15:04:01.024299 28394 mini_ranger_kms.cc:276] Ranger KMS PID: 15775
I20260809 15:04:01.024398 28394 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.186.148:39651
Aug 09, 2026 3:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 3:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 3:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 3:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 15:04:03 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786287843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.186.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 15:04:03 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.186.148@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:03.575253 28394 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.359s user 0.083s sys 0.288s
I20260809 15:04:03.575505 28394 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 15:04:03.575685 28394 mini_ranger_kms.cc:401] 127.27.186.148:39651/kms/v1/keys
Aug 09 15:04:03 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.186.148@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
Aug 09 15:04:06 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:04:06.111354 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:32881
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:32881
--ranger_config_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:06.451594 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:06.451838 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:06.451962 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:06.458925 15865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:06.459017 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:06.459087 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:06.459168 15865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:06.459231 15865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:06.468572 15865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.186.148:39651/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:32881
--ranger_config_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:06.469657 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:06.470850 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:06.481923 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:06.481926 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:06.481928 15878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:06.482370 15865 server_base.cc:1034] running on GCE node
I20260809 15:04:06.483285 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:06.484792 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:06.485984 15865 hybrid_clock.cc:648] HybridClock initialized: now 1786287846485942 us; error 31 us; skew 500 ppm
I20260809 15:04:06.486322 15865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:06 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:06.489060 15865 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:06.490432 15865 webserver.cc:460] Webserver started at http://127.27.186.190:37307/ using document root <none> and password file <none>
I20260809 15:04:06.490994 15865 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:06.491110 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:06.491390 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:04:06 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:06.547187 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "1569262e6342452a9740eee13ae17dca"
format_stamp: "Formatted at 2026-08-09 15:04:06 on dist-test-slave-q9h9"
server_key: "a957c96df456d8c97eb56622d4b32ee0"
server_key_iv: "f86d188a12d3cb264a81a1d6d2ea2286"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:06.548276 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "1569262e6342452a9740eee13ae17dca"
format_stamp: "Formatted at 2026-08-09 15:04:06 on dist-test-slave-q9h9"
server_key: "a957c96df456d8c97eb56622d4b32ee0"
server_key_iv: "f86d188a12d3cb264a81a1d6d2ea2286"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:06.555101 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 15:04:06.573462 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:06.575125 15865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:06.575320 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "1569262e6342452a9740eee13ae17dca"
format_stamp: "Formatted at 2026-08-09 15:04:06 on dist-test-slave-q9h9"
server_key: "a957c96df456d8c97eb56622d4b32ee0"
server_key_iv: "f86d188a12d3cb264a81a1d6d2ea2286"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:06.575541 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:06.591269 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:06.594548 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:06.594820 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:06.611759 15865 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:32881
I20260809 15:04:06.611784 15969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:32881 every 8 connection(s)
I20260809 15:04:06.612931 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:06.613332 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 15865
I20260809 15:04:06.613533 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:04:06.614328 28394 external_mini_cluster.cc:1428] Setting key 837de347de7cf2e3549f4c08fe9904ca
I20260809 15:04:06.616971 15971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 15:04:06 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:06.628683 15971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Bootstrap starting.
I20260809 15:04:06.631808 15971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:06.632938 15971 log.cc:824] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:06.635967 15971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: No bootstrap required, opened a new log
I20260809 15:04:06.635990 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:06.616120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.27.186.190:32881)
0809 15:04:06.616867 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:06.616882 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:06.616930 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:04:06.619063 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:06.619094 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:06.619533 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:06.619937 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:06.621165 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:06.623160 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:06.624371 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:06.625333 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:06.625499 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:06.629036 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:06.629047 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:06.629064 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:06.629112 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:06.632048 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:06.632814 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:06.632820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:06.632825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:06.632975 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:06.633392 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:06.633397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:06.633400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:06.634094 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:06.634363 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:06.634752 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 15:04:06.635028 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":318,"threads_started":1}
I20260809 15:04:06.643424 15971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:06.643723 15971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:06.643807 15971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1569262e6342452a9740eee13ae17dca, State: Initialized, Role: FOLLOWER
I20260809 15:04:06.644315 15971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:06.644527 15971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:06.644649 15971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:06.644807 15971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:06.646912 15971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:06.647334 15971 leader_election.cc:302] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1569262e6342452a9740eee13ae17dca; no voters:
I20260809 15:04:06.647863 15971 leader_election.cc:288] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:04:06.647957 15977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:06.648597 15977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 LEADER]: Becoming Leader. State: Replica: 1569262e6342452a9740eee13ae17dca, State: Running, Role: LEADER
I20260809 15:04:06.649091 15977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:06.649619 15971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:06.652484 15981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1569262e6342452a9740eee13ae17dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } } }
I20260809 15:04:06.652463 15986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1569262e6342452a9740eee13ae17dca. Latest consensus state: current_term: 1 leader_uuid: "1569262e6342452a9740eee13ae17dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } } }
I20260809 15:04:06.652859 15981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:06.652864 15986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:06.653995 16003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:06.659029 16003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:06.659587 15971 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 15:04:06.677537 16003 catalog_manager.cc:1347] Generated new cluster ID: 94522e36b7a04e2285a0ab05db015a37
I20260809 15:04:06.677660 16003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:06.714310 16003 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:04:06.716216 16003 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:06.728381 16003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Generated new TSK 0
I20260809 15:04:06.729377 16003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:04:07.433584 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:32881
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:04:07 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786287847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 15:04:07.823313 16051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:07.823565 16051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:07.823743 16051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:07.832641 16051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:07.832758 16051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:07.832924 16051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:04:07.843559 16051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.186.148:39651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:07.845228 16051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:07.846902 16051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:07.873327 16062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:07.874375 16063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:07.887580 16065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:07.888103 16051 server_base.cc:1034] running on GCE node
I20260809 15:04:07.888734 16051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:07.889753 16051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:07.891004 16051 hybrid_clock.cc:648] HybridClock initialized: now 1786287847890946 us; error 56 us; skew 500 ppm
I20260809 15:04:07.891317 16051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:07 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:07.894375 16051 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:04:07.896063 16051 webserver.cc:460] Webserver started at http://127.27.186.129:33657/ using document root <none> and password file <none>
I20260809 15:04:07.896735 16051 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:07.896855 16051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:07.897157 16051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:04:07 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:07.923254 16051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "7949abd036f0499a9a4987217a1f52d0"
format_stamp: "Formatted at 2026-08-09 15:04:07 on dist-test-slave-q9h9"
server_key: "e1dddf816f0e31ebda46503ad98b4cbd"
server_key_iv: "df18ac0d6283d6a18e49f6867078ab08"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:07.924464 16051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "7949abd036f0499a9a4987217a1f52d0"
format_stamp: "Formatted at 2026-08-09 15:04:07 on dist-test-slave-q9h9"
server_key: "e1dddf816f0e31ebda46503ad98b4cbd"
server_key_iv: "df18ac0d6283d6a18e49f6867078ab08"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:07.932186 16051 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260809 15:04:07.954388 16079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:07.955901 16051 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260809 15:04:07.956069 16051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "7949abd036f0499a9a4987217a1f52d0"
format_stamp: "Formatted at 2026-08-09 15:04:07 on dist-test-slave-q9h9"
server_key: "e1dddf816f0e31ebda46503ad98b4cbd"
server_key_iv: "df18ac0d6283d6a18e49f6867078ab08"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:07.956378 16051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:07.979390 16051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:07.983685 16051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:07.984071 16051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:07.994115 16051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:08.002254 16051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:08.002403 16051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:08.002605 16051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:08.002753 16051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 15:04:08 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:08.316785 16051 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:38931
I20260809 15:04:08.316819 16256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:38931 every 8 connection(s)
I20260809 15:04:08.318226 16051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:04:08.318845 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 16051
I20260809 15:04:08.319059 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:04:08.319928 28394 external_mini_cluster.cc:1428] Setting key cbf7f5ab45241bc1f06c7a10f3a16697
Aug 09 15:04:08 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:08.375613 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:08.335841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:52305 (local address 127.27.186.190:32881)
0809 15:04:08.355349 (+ 19508us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:08.355356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:08.355381 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:04:08.357192 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:08.357199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:08.357303 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:08.357395 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:08.359647 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:08.360824 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:08.363171 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:08.363835 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:08.363936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:08.368319 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:08.368326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:08.368330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:08.368371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:08.371934 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:08.372832 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:08.372838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:08.372842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:08.372989 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:08.373530 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:08.373535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:08.373539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:08.373912 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:08.374064 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:08.375038 (+ 974us) server_negotiation.cc:299] Negotiation successful
0809 15:04:08.375257 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19374,"thread_start_us":19305,"threads_started":1}
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
I20260809 15:04:08.377909 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:08.335345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:32881 (local address 127.27.186.129:52305)
0809 15:04:08.353125 (+ 17780us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:08.353187 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:08.356857 (+ 3670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:08.357968 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:08.357980 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:08.358621 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:08.359431 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:08.359458 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:08.361014 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:08.361019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:08.362997 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:08.363020 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:08.364286 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:08.364295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:08.364586 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:08.365397 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:08.365429 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:08.368068 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:08.372096 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:08.372106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:08.372121 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:08.372672 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:08.373137 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:08.373142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:08.373146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:08.373390 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:08.374165 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:08.374173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:08.374565 (+ 392us) client_negotiation.cc:769] Sending connection context
0809 15:04:08.376066 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0809 15:04:08.376551 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17417,"thread_start_us":17146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
I20260809 15:04:08.386188 16257 heartbeater.cc:347] Connected to a master server at 127.27.186.190:32881
I20260809 15:04:08.386579 16257 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:08.387543 16257 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
I20260809 15:04:08.390194 15911 ts_manager.cc:194] Registered new tserver with Master: 7949abd036f0499a9a4987217a1f52d0 (127.27.186.129:38931)
I20260809 15:04:08.392230 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:52305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:04:08.413848 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:32881
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:04:08 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
W20260809 15:04:08.852475 16266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:08.852736 16266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:08.852878 16266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:08.860203 16266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:08.860306 16266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:08.860424 16266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:04:08.869704 16266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.186.148:39651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:08.870837 16266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:08.872001 16266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:08.884549 16290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:08.884604 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:08.884565 16294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:08.885195 16266 server_base.cc:1034] running on GCE node
I20260809 15:04:08.885735 16266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:08.886637 16266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:08.887838 16266 hybrid_clock.cc:648] HybridClock initialized: now 1786287848887806 us; error 25 us; skew 500 ppm
I20260809 15:04:08.888120 16266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:08 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:08.895021 16266 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:04:08.896376 16266 webserver.cc:460] Webserver started at http://127.27.186.130:38409/ using document root <none> and password file <none>
I20260809 15:04:08.897092 16266 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:08.897256 16266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:08.897567 16266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:04:08 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:08.914036 16266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "341c1d2a78714b7fb04016f11336843f"
format_stamp: "Formatted at 2026-08-09 15:04:08 on dist-test-slave-q9h9"
server_key: "69daec4b620f327f69a837cf38e2a940"
server_key_iv: "adbe5072c504f2b27867dacb292fb278"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:08.915112 16266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "341c1d2a78714b7fb04016f11336843f"
format_stamp: "Formatted at 2026-08-09 15:04:08 on dist-test-slave-q9h9"
server_key: "69daec4b620f327f69a837cf38e2a940"
server_key_iv: "adbe5072c504f2b27867dacb292fb278"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:08.921689 16266 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 15:04:08.938598 16307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:08.940351 16266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 15:04:08.940550 16266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "341c1d2a78714b7fb04016f11336843f"
format_stamp: "Formatted at 2026-08-09 15:04:08 on dist-test-slave-q9h9"
server_key: "69daec4b620f327f69a837cf38e2a940"
server_key_iv: "adbe5072c504f2b27867dacb292fb278"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:08.940799 16266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:09.003831 16266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:09.007687 16266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:09.007953 16266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:09.008932 16266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:09.010375 16266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:09.010447 16266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:09.010562 16266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:09.010620 16266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:09.044009 16266 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:45823
I20260809 15:04:09.044045 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:45823 every 8 connection(s)
I20260809 15:04:09.045183 16266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:04:09.052111 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 16266
I20260809 15:04:09.052282 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:04:09.052809 28394 external_mini_cluster.cc:1428] Setting key 43f0c6614825185543821de512c8836a
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:09.065227 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.047724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56841 (local address 127.27.186.190:32881)
0809 15:04:09.048184 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:09.048188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:09.049261 (+ 1073us) server_negotiation.cc:407] Connection header received
0809 15:04:09.051099 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:09.051105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:09.051219 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:09.051326 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:09.053030 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.053901 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.055801 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.056224 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.056336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.059794 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:09.059799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:09.059802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:09.059833 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:09.062359 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.063050 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.063054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.063056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.063169 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.063599 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.063603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.063605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.063871 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:09.063999 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:09.064639 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 15:04:09.064858 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":258,"threads_started":1}
I20260809 15:04:09.065997 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.048284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:32881 (local address 127.27.186.130:56841)
0809 15:04:09.049093 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:09.049147 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:09.050834 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:09.051572 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:09.051579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:09.052095 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:09.052844 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:09.052870 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.054120 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.054125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:09.055625 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:09.055638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.056448 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.056455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:09.056667 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.057338 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:09.057367 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:09.059590 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:09.062509 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:09.062518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:09.062532 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:09.062921 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:09.063277 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:09.063282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:09.063286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:09.063477 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:09.063999 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:09.064007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:09.064286 (+ 279us) client_negotiation.cc:769] Sending connection context
0809 15:04:09.064497 (+ 211us) client_negotiation.cc:240] Negotiation successful
0809 15:04:09.064943 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":320,"threads_started":1}
I20260809 15:04:09.067379 16503 heartbeater.cc:347] Connected to a master server at 127.27.186.190:32881
I20260809 15:04:09.067631 16503 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:09.068239 16503 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
I20260809 15:04:09.069645 15911 ts_manager.cc:194] Registered new tserver with Master: 341c1d2a78714b7fb04016f11336843f (127.27.186.130:45823)
I20260809 15:04:09.070688 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:56841
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:04:09.105448 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:32881
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:09.395764 16257 heartbeater.cc:503] Master 127.27.186.190:32881 was elected leader, sending a full tablet report...
W20260809 15:04:09.420634 16513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:09.420881 16513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:09.421010 16513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:09.427907 16513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:09.428004 16513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:09.428154 16513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:04:09.437196 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.186.148:39651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:09.438421 16513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:09.439582 16513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:09.451666 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:09.451728 16528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:09.451625 16520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:09.452687 16513 server_base.cc:1034] running on GCE node
I20260809 15:04:09.453202 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:09.454056 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:09.455266 16513 hybrid_clock.cc:648] HybridClock initialized: now 1786287849455225 us; error 29 us; skew 500 ppm
I20260809 15:04:09.455554 16513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287849, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:09.458484 16513 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:04:09.459846 16513 webserver.cc:460] Webserver started at http://127.27.186.131:42713/ using document root <none> and password file <none>
I20260809 15:04:09.460479 16513 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:09.460580 16513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:09.460837 16513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287849, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:09.473193 16513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "9f8c3a226f6a48d8be7b75a725cbda7d"
format_stamp: "Formatted at 2026-08-09 15:04:09 on dist-test-slave-q9h9"
server_key: "a056a3afeb873772682aaf1a39718477"
server_key_iv: "38230a4c7c2550544c1ab49650ed81a0"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:09.473938 16513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "9f8c3a226f6a48d8be7b75a725cbda7d"
format_stamp: "Formatted at 2026-08-09 15:04:09 on dist-test-slave-q9h9"
server_key: "a056a3afeb873772682aaf1a39718477"
server_key_iv: "38230a4c7c2550544c1ab49650ed81a0"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:09.478973 16513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 15:04:09.493649 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:09.494995 16513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:04:09.495124 16513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "9f8c3a226f6a48d8be7b75a725cbda7d"
format_stamp: "Formatted at 2026-08-09 15:04:09 on dist-test-slave-q9h9"
server_key: "a056a3afeb873772682aaf1a39718477"
server_key_iv: "38230a4c7c2550544c1ab49650ed81a0"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:09.495317 16513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:09.513383 16513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:09.516690 16513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:09.516937 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:09.517776 16513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:09.519106 16513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:09.519186 16513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:09.519296 16513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:09.519345 16513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:09.549494 16513 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:33627
I20260809 15:04:09.549526 16723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:33627 every 8 connection(s)
I20260809 15:04:09.550729 16513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:04:09.554181 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 16513
I20260809 15:04:09.554370 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:04:09.555029 28394 external_mini_cluster.cc:1428] Setting key 8a7c8985c1ad1d5842008530135bae5d
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287849, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:09.569154 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.553023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:56901 (local address 127.27.186.190:32881)
0809 15:04:09.553227 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:09.553233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:09.554344 (+ 1111us) server_negotiation.cc:407] Connection header received
0809 15:04:09.556677 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:09.556681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:09.556764 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:09.556848 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:09.558423 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.559236 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.560776 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.561206 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.561303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.564224 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:09.564229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:09.564232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:09.564265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:09.566560 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.567195 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.567199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.567201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.567323 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.567673 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.567676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.567678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.567923 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:09.568027 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:09.568626 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 15:04:09.568816 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:04:09.569972 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.553403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:32881 (local address 127.27.186.131:56901)
0809 15:04:09.554179 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:09.554228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:09.556405 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:09.557080 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:09.557090 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:09.557534 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:09.558244 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:09.558263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.559406 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.559412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:09.560655 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:09.560667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.561389 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.561394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:09.561567 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.562155 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:09.562176 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:09.564043 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:09.566687 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:09.566694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:09.566704 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:09.567081 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:09.567405 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:09.567408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:09.567410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:09.567584 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:09.568014 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:09.568019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:09.568262 (+ 243us) client_negotiation.cc:769] Sending connection context
0809 15:04:09.568474 (+ 212us) client_negotiation.cc:240] Negotiation successful
0809 15:04:09.568858 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":286,"threads_started":1}
I20260809 15:04:09.571264 16724 heartbeater.cc:347] Connected to a master server at 127.27.186.190:32881
I20260809 15:04:09.571532 16724 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:09.572142 16724 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
I20260809 15:04:09.573401 15911 ts_manager.cc:194] Registered new tserver with Master: 9f8c3a226f6a48d8be7b75a725cbda7d (127.27.186.131:33627)
I20260809 15:04:09.574188 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:56901
I20260809 15:04:09.582481 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:04:09.584134 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 16051
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:04:09.599431 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.585119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.27.186.129:38931)
0809 15:04:09.585754 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:09.585760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:09.585785 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:04:09.585837 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:09.585843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:09.586004 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:09.586165 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:09.587572 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.588554 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.589916 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.590479 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.590573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.594458 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:09.594469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:09.594480 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:09.594516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:09.596980 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.597599 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.597605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.597608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.597747 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.598064 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.598069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.598071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.598369 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:09.598600 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:09.598882 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 15:04:09.599087 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
I20260809 15:04:09.600550 16248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:09.683171 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 16051
I20260809 15:04:09.700018 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 16266
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:04:09.715766 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.700923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.27.186.130:45823)
0809 15:04:09.701444 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:09.701451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:09.701476 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:04:09.701533 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:09.701539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:09.701729 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:09.701848 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:09.703074 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.704119 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.705440 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.705961 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.706118 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.710072 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:09.710083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:09.710095 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:09.710126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:09.712699 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.713563 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.713572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.713576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.713748 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.714200 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.714205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.714209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.714533 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:09.714821 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:09.715148 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 15:04:09.715401 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260809 15:04:09.716921 16489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:09.796627 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 16266
I20260809 15:04:09.819733 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 16513
Aug 09 15:04:09 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:04:09.835034 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:09.820602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.27.186.131:33627)
0809 15:04:09.821198 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:09.821205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:09.821242 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:04:09.821302 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:09.821309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:09.821519 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:09.821644 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:09.822959 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.824010 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.825290 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:09.825838 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:09.825935 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:09.829961 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:09.829972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:09.829985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:09.830047 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:09.832641 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.833275 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.833280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.833283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.833407 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:09.833712 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:09.833716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:09.833718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:09.834060 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:09.834324 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:09.834555 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 15:04:09.834741 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":292,"threads_started":1}
I20260809 15:04:09.836213 16717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:09.916774 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 16513
I20260809 15:04:09.936558 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 15865
I20260809 15:04:09.937695 15964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:10.041438 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 15865
I20260809 15:04:10.074255 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:32881
--webserver_interface=127.27.186.190
--webserver_port=37307
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:32881
--ranger_config_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:10.400398 16739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:10.400673 16739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:10.400818 16739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:10.407915 16739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:10.408031 16739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:10.408124 16739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:10.408207 16739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:10.408282 16739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:10.417940 16739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:32881
--ranger_config_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:10.419071 16739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:10.420382 16739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:10.433314 16751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:10.433348 16748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:10.433516 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:10.434353 16739 server_base.cc:1034] running on GCE node
I20260809 15:04:10.435091 16739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:10.436408 16739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:10.437628 16739 hybrid_clock.cc:648] HybridClock initialized: now 1786287850437566 us; error 47 us; skew 500 ppm
I20260809 15:04:10.437958 16739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:10 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287850, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:10.440886 16739 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:10.442384 16739 webserver.cc:460] Webserver started at http://127.27.186.190:37307/ using document root <none> and password file <none>
I20260809 15:04:10.442993 16739 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:10.443138 16739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 15:04:10 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287850, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:10.528908 16739 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 15:04:10.532465 16763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:10.534103 16739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:10.534317 16739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "1569262e6342452a9740eee13ae17dca"
format_stamp: "Formatted at 2026-08-09 15:04:06 on dist-test-slave-q9h9"
server_key: "a957c96df456d8c97eb56622d4b32ee0"
server_key_iv: "f86d188a12d3cb264a81a1d6d2ea2286"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:10.535640 16739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:10.553332 16739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:10.557996 16739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:10.558341 16739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:10.579674 16739 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:32881
I20260809 15:04:10.579640 16843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:32881 every 8 connection(s)
I20260809 15:04:10.581426 16739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:10.587539 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 16739
I20260809 15:04:10.588402 16844 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:04:10.588593 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:38931
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:32881
--webserver_port=33657
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:10.592593 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Bootstrap starting.
I20260809 15:04:10.612072 16844 log.cc:824] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:10.620656 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:04:10.621141 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Bootstrap complete.
I20260809 15:04:10.630119 16844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:10.630925 16844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1569262e6342452a9740eee13ae17dca, State: Initialized, Role: FOLLOWER
I20260809 15:04:10.631575 16844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:10.631786 16844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:10.631910 16844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:10.632078 16844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:04:10.635776 16844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:10.636229 16844 leader_election.cc:302] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1569262e6342452a9740eee13ae17dca; no voters:
I20260809 15:04:10.636777 16844 leader_election.cc:288] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 15:04:10.636973 16849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:04:10.640964 16849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [term 2 LEADER]: Becoming Leader. State: Replica: 1569262e6342452a9740eee13ae17dca, State: Running, Role: LEADER
I20260809 15:04:10.641649 16849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } }
I20260809 15:04:10.645617 16844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:10.651427 16852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1569262e6342452a9740eee13ae17dca. Latest consensus state: current_term: 2 leader_uuid: "1569262e6342452a9740eee13ae17dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } } }
I20260809 15:04:10.651952 16852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:10.658623 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1569262e6342452a9740eee13ae17dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1569262e6342452a9740eee13ae17dca" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 32881 } } }
I20260809 15:04:10.658905 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:10.659559 16858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:10.666460 16858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:10.667917 16858 catalog_manager.cc:1259] Loaded cluster ID: 94522e36b7a04e2285a0ab05db015a37
I20260809 15:04:10.668113 16858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:10.672032 16858 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:10.673415 16858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Loaded TSK: 0
I20260809 15:04:10.679277 16858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:04:10.720814 16844 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260809 15:04:11.069005 16846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:11.069500 16846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:11.069696 16846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:11.081002 16846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:11.081215 16846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:11.081461 16846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:04:11.095330 16846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:38931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=33657
--tserver_master_addrs=127.27.186.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:11.096942 16846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:11.098632 16846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:11.125826 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:11.125847 16901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:11.137395 16903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:11.138253 16846 server_base.cc:1034] running on GCE node
I20260809 15:04:11.138895 16846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:11.139935 16846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:11.141146 16846 hybrid_clock.cc:648] HybridClock initialized: now 1786287851141059 us; error 82 us; skew 500 ppm
I20260809 15:04:11.141461 16846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:11 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:11.144796 16846 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:04:11.147197 16846 webserver.cc:460] Webserver started at http://127.27.186.129:33657/ using document root <none> and password file <none>
I20260809 15:04:11.147876 16846 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:11.148015 16846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 15:04:11 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:11.184094 16846 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.000s
I20260809 15:04:11.193171 16918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:11.195182 16846 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.002s
I20260809 15:04:11.195377 16846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "7949abd036f0499a9a4987217a1f52d0"
format_stamp: "Formatted at 2026-08-09 15:04:07 on dist-test-slave-q9h9"
server_key: "e1dddf816f0e31ebda46503ad98b4cbd"
server_key_iv: "df18ac0d6283d6a18e49f6867078ab08"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:11.196355 16846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:11.226944 16846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:11.231609 16846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:11.231945 16846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:11.235198 16846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:11.236599 16846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:11.236698 16846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:11.236806 16846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:11.236883 16846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:11.290401 16846 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:38931
I20260809 15:04:11.290481 17110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:38931 every 8 connection(s)
I20260809 15:04:11.292366 16846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:04:11.294677 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 16846
I20260809 15:04:11.295521 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:45823
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:32881
--webserver_port=38409
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:04:11 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:11.329465 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:11.295989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:32881 (local address 127.27.186.129:51635)
0809 15:04:11.298358 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:11.298408 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:11.304660 (+ 6252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:11.306850 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:11.306861 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:11.307398 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:11.308191 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:11.308216 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.312191 (+ 3975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.312197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:11.314555 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:11.314572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.316225 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.316235 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:11.316504 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:11.317249 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:11.317284 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:11.319978 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:11.324492 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:11.324501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:11.324515 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:11.325092 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:11.325664 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:11.325669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:11.325672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:11.325940 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:11.326923 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:11.326931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:11.327302 (+ 371us) client_negotiation.cc:769] Sending connection context
0809 15:04:11.327601 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 15:04:11.328129 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"thread_start_us":1850,"threads_started":1}
I20260809 15:04:11.332510 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:11.297192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:51635 (local address 127.27.186.190:32881)
0809 15:04:11.301930 (+ 4738us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:11.301946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:11.301994 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:04:11.305389 (+ 3395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:11.305406 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:11.305829 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:11.306225 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:11.308382 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.310156 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.314879 (+ 4723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.315949 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.316960 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:11.320288 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:11.320299 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:11.320317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:11.320366 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:11.324327 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:11.325293 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:11.325301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:11.325309 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:11.325520 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:11.326127 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:11.326131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:11.326135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:11.326758 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:11.330629 (+ 3871us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:11.330894 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 15:04:11.331271 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4510,"thread_start_us":4278,"threads_started":1}
I20260809 15:04:11.336236 17112 heartbeater.cc:347] Connected to a master server at 127.27.186.190:32881
I20260809 15:04:11.336558 17112 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:11.337459 17112 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
W20260809 15:04:11.340806 17112 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:32881 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 15:04:11.341027 17112 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:11.341310 17112 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
I20260809 15:04:11.342682 17112 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:11.343254 17112 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
W20260809 15:04:11.344384 17112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 15:04:11.718010 17117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:11.718427 17117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:11.718614 17117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:11.725593 17117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:11.725698 17117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:11.725884 17117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:04:11.735157 17117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:45823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=38409
--tserver_master_addrs=127.27.186.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:11.736295 17117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:11.737521 17117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:11.751662 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:11.752265 17138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:11.752794 17117 server_base.cc:1034] running on GCE node
W20260809 15:04:11.752993 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:11.753619 17117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:11.754627 17117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:11.755842 17117 hybrid_clock.cc:648] HybridClock initialized: now 1786287851755771 us; error 61 us; skew 500 ppm
I20260809 15:04:11.756196 17117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:11 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:11.759532 17117 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:04:11.761183 17117 webserver.cc:460] Webserver started at http://127.27.186.130:38409/ using document root <none> and password file <none>
I20260809 15:04:11.762079 17117 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:11.762243 17117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 15:04:11 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:11.787863 17117 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260809 15:04:11.790640 17156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:11.791993 17117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:11.792145 17117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "341c1d2a78714b7fb04016f11336843f"
format_stamp: "Formatted at 2026-08-09 15:04:08 on dist-test-slave-q9h9"
server_key: "69daec4b620f327f69a837cf38e2a940"
server_key_iv: "adbe5072c504f2b27867dacb292fb278"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:11.793093 17117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:11.810468 17117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:11.814541 17117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:11.814918 17117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:11.816490 17117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:11.818805 17117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:11.818917 17117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:11.819022 17117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:11.819101 17117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:11.860546 17117 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:45823
I20260809 15:04:11.860601 17347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:45823 every 8 connection(s)
I20260809 15:04:11.862444 17117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:04:11.869272 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 17117
I20260809 15:04:11.870100 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.186.148:39651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:33627
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:32881
--webserver_port=42713
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:04:11 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:11.900874 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:11.865688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:32881 (local address 127.27.186.130:38425)
0809 15:04:11.881374 (+ 15686us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:11.881422 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:11.883631 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:11.884489 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:11.884497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:11.884892 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:11.885452 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:11.885468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.886605 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.886609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:11.887872 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:11.887885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.888728 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.888736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:11.888961 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:11.889627 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:11.889648 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:11.892472 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:11.896444 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:11.896453 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:11.896468 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:11.897019 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:11.897486 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:11.897492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:11.897496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:11.897713 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:11.898402 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:11.898410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:11.898773 (+ 363us) client_negotiation.cc:769] Sending connection context
0809 15:04:11.899046 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 15:04:11.899568 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15402,"thread_start_us":15204,"threads_started":1}
I20260809 15:04:11.902668 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:11.865043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:38425 (local address 127.27.186.190:32881)
0809 15:04:11.880845 (+ 15802us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:11.880851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:11.883996 (+ 3145us) server_negotiation.cc:407] Connection header received
0809 15:04:11.884058 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:11.884061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:11.884127 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:11.884219 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:11.885632 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.886449 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.888017 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:11.888489 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:11.888636 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:11.892726 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:11.892732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:11.892737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:11.892778 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:11.896286 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:11.897193 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:11.897198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:11.897202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:11.897356 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:11.897853 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:11.897858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:11.897862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:11.898252 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:11.901928 (+ 3676us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:11.902090 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 15:04:11.902318 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15669,"thread_start_us":18835,"threads_started":1}
I20260809 15:04:11.904044 17351 heartbeater.cc:347] Connected to a master server at 127.27.186.190:32881
I20260809 15:04:11.904379 17351 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:11.905232 17351 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
I20260809 15:04:11.907296 16793 ts_manager.cc:194] Registered new tserver with Master: 341c1d2a78714b7fb04016f11336843f (127.27.186.130:45823)
I20260809 15:04:11.909430 16793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:38425
Aug 09 15:04:12 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786287852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 15:04:12.287227 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:12.287614 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:12.287770 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:12.304714 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:12.304857 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:12.305094 17355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:04:12.319993 17355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.186.148:39651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:33627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=42713
--tserver_master_addrs=127.27.186.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:12.321667 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:12.323441 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:12.344918 17363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:12.345702 17112 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:12.346124 17112 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
W20260809 15:04:12.346928 17367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:12.347345 16793 ts_manager.cc:194] Registered new tserver with Master: 7949abd036f0499a9a4987217a1f52d0 (127.27.186.129:38931)
I20260809 15:04:12.348521 16793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:51635
W20260809 15:04:12.359236 17372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:12.359972 17355 server_base.cc:1034] running on GCE node
I20260809 15:04:12.360702 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:12.362299 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:12.363597 17355 hybrid_clock.cc:648] HybridClock initialized: now 1786287852363545 us; error 82 us; skew 500 ppm
I20260809 15:04:12.363994 17355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:12 dist-test-slave-q9h9 krb5kdc[15151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:12.368206 17355 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:04:12.372238 17355 webserver.cc:460] Webserver started at http://127.27.186.131:42713/ using document root <none> and password file <none>
I20260809 15:04:12.373235 17355 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:12.373466 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 15:04:12 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:12.426316 17355 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.003s sys 0.007s
I20260809 15:04:12.436084 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:12.438043 17355 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260809 15:04:12.438261 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "9f8c3a226f6a48d8be7b75a725cbda7d"
format_stamp: "Formatted at 2026-08-09 15:04:09 on dist-test-slave-q9h9"
server_key: "a056a3afeb873772682aaf1a39718477"
server_key_iv: "38230a4c7c2550544c1ab49650ed81a0"
server_key_version: "kuduclusterkey@0"
I20260809 15:04:12.439812 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 15:04:12 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:12.536186 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:12.541014 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:12.541296 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:12.542443 17355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:12.544207 17355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:12.544332 17355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:12.544471 17355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:12.544554 17355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:12.596835 17355 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:33627
I20260809 15:04:12.596932 17579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:33627 every 8 connection(s)
I20260809 15:04:12.598582 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:04:12.599231 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 17355
I20260809 15:04:12.627646 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.600297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.27.186.190:32881)
0809 15:04:12.604225 (+ 3928us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.604231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.605077 (+ 846us) server_negotiation.cc:407] Connection header received
0809 15:04:12.605148 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.605154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.605233 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.605338 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:12.608038 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.609258 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.614625 (+ 5367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.615302 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.615415 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.620000 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:12.620006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:12.620011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:12.620045 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:12.623542 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:12.624342 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:12.624347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:12.624351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:12.624503 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:12.624962 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:12.624967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:12.624970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:12.625335 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:12.625607 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.626783 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.626997 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3799,"thread_start_us":3663,"threads_started":1}
Aug 09 15:04:12 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:12.637444 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.606341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:54677 (local address 127.27.186.190:32881)
0809 15:04:12.619459 (+ 13118us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.619465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.619482 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:04:12.619528 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.619533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.619600 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.619705 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:12.621741 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.623106 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.625439 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.626135 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.626255 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.630500 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:12.630507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:12.630512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:12.630547 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:12.633830 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:12.634774 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:12.634780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:12.634783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:12.634943 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:12.635516 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:12.635521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:12.635524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:12.635863 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:12.635993 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.636870 (+ 877us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.637063 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12997,"thread_start_us":12870,"threads_started":1}
I20260809 15:04:12.638495 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.606915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:32881 (local address 127.27.186.131:54677)
0809 15:04:12.612496 (+ 5581us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:12.612556 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:12.614960 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:12.620025 (+ 5065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:12.620037 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:12.620649 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:12.621461 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.621488 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.623305 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.623310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.625267 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.625282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.626551 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.626560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.626847 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.627530 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:12.627559 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:12.630216 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:12.634009 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:12.634043 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:12.634058 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:12.634631 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:12.635097 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:12.635102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:12.635106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:12.635364 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:12.636036 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:12.636044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:12.636413 (+ 369us) client_negotiation.cc:769] Sending connection context
0809 15:04:12.636741 (+ 328us) client_negotiation.cc:240] Negotiation successful
0809 15:04:12.637238 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5234,"thread_start_us":11010,"threads_started":1}
I20260809 15:04:12.639994 17580 heartbeater.cc:347] Connected to a master server at 127.27.186.190:32881
I20260809 15:04:12.640318 17580 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:12.641175 17580 heartbeater.cc:511] Master 127.27.186.190:32881 requested a full tablet report, sending...
I20260809 15:04:12.642899 16793 ts_manager.cc:194] Registered new tserver with Master: 9f8c3a226f6a48d8be7b75a725cbda7d (127.27.186.131:33627)
I20260809 15:04:12.643762 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:04:12.644083 16793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:54677
I20260809 15:04:12.667675 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.652427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.27.186.190:32881)
0809 15:04:12.652584 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.652589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.652796 (+ 207us) server_negotiation.cc:407] Connection header received
0809 15:04:12.653072 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.653078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.653149 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.653242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:12.655257 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.656323 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.658008 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.658672 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.658805 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.662100 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:12.662106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:12.662111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:12.662149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:12.665071 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:12.665724 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:12.665728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:12.665730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:12.665841 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:12.666264 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:12.666268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:12.666270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:12.666586 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:12.666811 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.667152 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.667351 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:04:12.672197 16793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:04:12.679466 16793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:04:12.716894 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.695236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:45823 (local address 127.0.0.1:52796)
0809 15:04:12.702477 (+ 7241us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:12.702515 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:12.702689 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:12.705364 (+ 2675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:12.705369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:12.705394 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:12.705708 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.705722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.709445 (+ 3723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.709451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.711952 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.711970 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.715971 (+ 4001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.715978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.716173 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.716230 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 15:04:12.716362 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 15:04:12.716543 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7127,"thread_start_us":7000,"threads_started":1}
I20260809 15:04:12.717760 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.696166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.27.186.130:45823)
0809 15:04:12.702095 (+ 5929us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.702104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.702898 (+ 794us) server_negotiation.cc:407] Connection header received
0809 15:04:12.702999 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.703007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.703243 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.703430 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:12.707715 (+ 4285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.709262 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.712153 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.713514 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.713691 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.713873 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.717263 (+ 3390us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.717435 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5750,"thread_start_us":6346,"threads_started":1}
I20260809 15:04:12.720220 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.695048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:38931 (local address 127.0.0.1:59158)
0809 15:04:12.701643 (+ 6595us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:12.703647 (+ 2004us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:12.703778 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:12.704762 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:12.704783 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:12.704815 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:12.705171 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.705190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.711526 (+ 6336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.711531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.713800 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.713815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.719492 (+ 5677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.719498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.719663 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.719696 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:04:12.719799 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 15:04:12.719918 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6337,"thread_start_us":6191,"threads_started":1}
I20260809 15:04:12.721001 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.695704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.27.186.129:38931)
0809 15:04:12.701085 (+ 5381us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.701094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.703900 (+ 2806us) server_negotiation.cc:407] Connection header received
0809 15:04:12.704080 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.704088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.704306 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.704482 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:12.705911 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.707529 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.713966 (+ 6437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.715455 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.715644 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.715817 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.720509 (+ 4692us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.720676 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5201,"thread_start_us":5024,"threads_started":1}
I20260809 15:04:12.722441 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.702644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:33627 (local address 127.0.0.1:53050)
0809 15:04:12.710167 (+ 7523us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:12.710183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:12.710291 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:12.711237 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:12.711241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:12.711263 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:12.711720 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.711732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.714699 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.714704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.717735 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.717751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.721719 (+ 3968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.721725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.721879 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.721909 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 15:04:12.722005 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 15:04:12.722146 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7395,"thread_start_us":7245,"threads_started":1}
I20260809 15:04:12.722682 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.702379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.27.186.131:33627)
0809 15:04:12.710828 (+ 8449us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.710834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.710856 (+ 22us) server_negotiation.cc:407] Connection header received
0809 15:04:12.710923 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.710928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.711061 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.711213 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:12.711957 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.714538 (+ 2581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.717920 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.721220 (+ 3300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.721410 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.721556 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.722124 (+ 568us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.722316 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8303,"thread_start_us":8134,"threads_started":1}
I20260809 15:04:12.722467 17247 tablet_service.cc:1467] Processing CreateTablet for tablet 1b21400d184340dea6adb60d921dc900 (DEFAULT_TABLE table=test-table [id=cf5b63c6d99f4c02a3b2725184a5d40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:12.724754 17247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b21400d184340dea6adb60d921dc900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:12.726347 16991 tablet_service.cc:1467] Processing CreateTablet for tablet 1b21400d184340dea6adb60d921dc900 (DEFAULT_TABLE table=test-table [id=cf5b63c6d99f4c02a3b2725184a5d40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:12.726804 17463 tablet_service.cc:1467] Processing CreateTablet for tablet 1b21400d184340dea6adb60d921dc900 (DEFAULT_TABLE table=test-table [id=cf5b63c6d99f4c02a3b2725184a5d40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:12.727546 16991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b21400d184340dea6adb60d921dc900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:12.728152 17463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b21400d184340dea6adb60d921dc900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:12.746194 17617 tablet_bootstrap.cc:492] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Bootstrap starting.
I20260809 15:04:12.751896 17617 tablet_bootstrap.cc:654] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:12.752640 17618 tablet_bootstrap.cc:492] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Bootstrap starting.
I20260809 15:04:12.754168 17617 log.cc:824] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:12.755247 17619 tablet_bootstrap.cc:492] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Bootstrap starting.
I20260809 15:04:12.756786 17618 tablet_bootstrap.cc:654] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:12.758067 17618 log.cc:824] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:12.758389 17619 tablet_bootstrap.cc:654] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:12.759574 17619 log.cc:824] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:12.763839 17617 tablet_bootstrap.cc:492] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: No bootstrap required, opened a new log
I20260809 15:04:12.764154 17618 tablet_bootstrap.cc:492] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: No bootstrap required, opened a new log
I20260809 15:04:12.764178 17617 ts_tablet_manager.cc:1397] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20260809 15:04:12.764417 17618 ts_tablet_manager.cc:1397] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260809 15:04:12.775557 17618 raft_consensus.cc:348] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.776085 17618 raft_consensus.cc:374] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:12.776226 17618 raft_consensus.cc:729] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8c3a226f6a48d8be7b75a725cbda7d, State: Initialized, Role: FOLLOWER
I20260809 15:04:12.776618 17619 tablet_bootstrap.cc:492] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: No bootstrap required, opened a new log
I20260809 15:04:12.776376 17617 raft_consensus.cc:348] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.776752 17617 raft_consensus.cc:374] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:12.776880 17617 raft_consensus.cc:729] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341c1d2a78714b7fb04016f11336843f, State: Initialized, Role: FOLLOWER
I20260809 15:04:12.777006 17619 ts_tablet_manager.cc:1397] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20260809 15:04:12.776996 17618 consensus_queue.cc:260] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.777576 17617 consensus_queue.cc:260] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
Aug 09 15:04:12 dist-test-slave-q9h9 krb5kdc[15151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for HTTP/127.27.186.148@KRBTEST.COM
I20260809 15:04:12.785558 17617 ts_tablet_manager.cc:1428] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Time spent starting tablet: real 0.021s user 0.012s sys 0.006s
I20260809 15:04:12.785925 17351 heartbeater.cc:503] Master 127.27.186.190:32881 was elected leader, sending a full tablet report...
I20260809 15:04:12.787642 17619 raft_consensus.cc:348] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.788618 17580 heartbeater.cc:503] Master 127.27.186.190:32881 was elected leader, sending a full tablet report...
I20260809 15:04:12.789222 17618 ts_tablet_manager.cc:1428] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Time spent starting tablet: real 0.025s user 0.016s sys 0.008s
I20260809 15:04:12.789827 17619 raft_consensus.cc:374] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:12.789984 17619 raft_consensus.cc:729] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7949abd036f0499a9a4987217a1f52d0, State: Initialized, Role: FOLLOWER
I20260809 15:04:12.795104 17619 consensus_queue.cc:260] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.796939 17112 heartbeater.cc:503] Master 127.27.186.190:32881 was elected leader, sending a full tablet report...
I20260809 15:04:12.797533 17619 ts_tablet_manager.cc:1428] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Time spent starting tablet: real 0.020s user 0.013s sys 0.000s
W20260809 15:04:12.805805 17113 tablet.cc:2369] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:12.855525 17624 raft_consensus.cc:482] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:12.855868 17624 raft_consensus.cc:504] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.858129 17624 leader_election.cc:288] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7949abd036f0499a9a4987217a1f52d0 (127.27.186.129:38931), 341c1d2a78714b7fb04016f11336843f (127.27.186.130:45823)
W20260809 15:04:12.864866 17585 tablet.cc:2369] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:12.868225 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.858483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:38931 (local address 127.27.186.131:40163)
0809 15:04:12.858948 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:12.858969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:12.860256 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:12.860749 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:12.860755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:12.860798 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:12.860993 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.861006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.862509 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.862515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.864468 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.864485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.867481 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.867488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.867675 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.867712 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 15:04:12.867832 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 15:04:12.867967 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338}
W20260809 15:04:12.870510 17352 tablet.cc:2369] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:12.870244 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.860089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:40163 (local address 127.27.186.129:38931)
0809 15:04:12.860272 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.860278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.860297 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:04:12.860393 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.860397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.860476 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.860564 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:12.861184 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.862325 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.865855 (+ 3530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.867128 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.867294 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.867371 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.869843 (+ 2472us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.869981 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 15:04:12.872403 17024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b21400d184340dea6adb60d921dc900" candidate_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7949abd036f0499a9a4987217a1f52d0" is_pre_election: true
I20260809 15:04:12.872982 17024 raft_consensus.cc:2454] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f8c3a226f6a48d8be7b75a725cbda7d in term 0.
I20260809 15:04:12.873878 17385 leader_election.cc:302] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7949abd036f0499a9a4987217a1f52d0, 9f8c3a226f6a48d8be7b75a725cbda7d; no voters:
I20260809 15:04:12.874477 17624 raft_consensus.cc:2781] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:04:12.874624 17624 raft_consensus.cc:482] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:04:12.874729 17624 raft_consensus.cc:3037] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:12.878475 17624 raft_consensus.cc:504] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.879559 17024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b21400d184340dea6adb60d921dc900" candidate_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7949abd036f0499a9a4987217a1f52d0"
I20260809 15:04:12.879772 17024 raft_consensus.cc:3037] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:12.880151 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.859412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37021 (local address 127.27.186.130:45823)
0809 15:04:12.859632 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.859638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.869227 (+ 9589us) server_negotiation.cc:407] Connection header received
0809 15:04:12.873080 (+ 3853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.873086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.873161 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.873221 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:12.873997 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.875100 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.877729 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.879037 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.879160 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.879228 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.879777 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.879914 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:04:12.880865 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.858752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:45823 (local address 127.27.186.131:37021)
0809 15:04:12.869109 (+ 10357us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:12.869133 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:12.872903 (+ 3770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:12.873411 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:12.873416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:12.873443 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:12.873809 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.873822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.875307 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.875312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.877534 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:12.877549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.879403 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.879409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:12.879585 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.879617 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:04:12.880368 (+ 751us) client_negotiation.cc:240] Negotiation successful
0809 15:04:12.880505 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10236,"thread_start_us":13165,"threads_started":1}
I20260809 15:04:12.881798 17276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b21400d184340dea6adb60d921dc900" candidate_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341c1d2a78714b7fb04016f11336843f" is_pre_election: true
I20260809 15:04:12.882326 17276 raft_consensus.cc:2454] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f8c3a226f6a48d8be7b75a725cbda7d in term 0.
I20260809 15:04:12.882964 17024 raft_consensus.cc:2454] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f8c3a226f6a48d8be7b75a725cbda7d in term 1.
I20260809 15:04:12.883554 17385 leader_election.cc:302] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7949abd036f0499a9a4987217a1f52d0, 9f8c3a226f6a48d8be7b75a725cbda7d; no voters:
I20260809 15:04:12.884883 17628 raft_consensus.cc:2781] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:12.885352 17276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b21400d184340dea6adb60d921dc900" candidate_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341c1d2a78714b7fb04016f11336843f"
I20260809 15:04:12.885552 17276 raft_consensus.cc:3037] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:12.886103 17624 leader_election.cc:288] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [CANDIDATE]: Term 1 election: Requested vote from peers 7949abd036f0499a9a4987217a1f52d0 (127.27.186.129:38931), 341c1d2a78714b7fb04016f11336843f (127.27.186.130:45823)
I20260809 15:04:12.888700 17276 raft_consensus.cc:2454] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f8c3a226f6a48d8be7b75a725cbda7d in term 1.
I20260809 15:04:12.898324 17628 raft_consensus.cc:686] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 1 LEADER]: Becoming Leader. State: Replica: 9f8c3a226f6a48d8be7b75a725cbda7d, State: Running, Role: LEADER
I20260809 15:04:12.898937 17628 consensus_queue.cc:237] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } }
I20260809 15:04:12.904958 16793 catalog_manager.cc:5495] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f8c3a226f6a48d8be7b75a725cbda7d (127.27.186.131). New cstate: current_term: 1 leader_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f8c3a226f6a48d8be7b75a725cbda7d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 33627 } health_report { overall_health: HEALTHY } } }
I20260809 15:04:12.973784 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:12.968393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.27.186.131:33627)
0809 15:04:12.968610 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:12.968617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:12.968654 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:04:12.968848 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:12.968853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:12.968931 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:12.969061 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:04:12.969451 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.970286 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.971675 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:12.972068 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:12.972178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:12.972661 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:04:12.973197 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:04:12.973340 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:12.973446 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 15:04:12.973525 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 15:04:12.981550 17276 raft_consensus.cc:1261] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 1 FOLLOWER]: Refusing update from remote peer 9f8c3a226f6a48d8be7b75a725cbda7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:12.981673 17024 raft_consensus.cc:1261] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 1 FOLLOWER]: Refusing update from remote peer 9f8c3a226f6a48d8be7b75a725cbda7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:12.982712 17629 consensus_queue.cc:1035] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7949abd036f0499a9a4987217a1f52d0" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:12.983131 17628 consensus_queue.cc:1035] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "341c1d2a78714b7fb04016f11336843f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 45823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:13.007879 17637 mvcc.cc:205] Tried to move back new op lower bound from 7316635045802397696 to 7316635045479571456. Current Snapshot: MvccSnapshot[applied={T|T < 7316635045802397696}]
I20260809 15:04:13.008947 17639 mvcc.cc:205] Tried to move back new op lower bound from 7316635045802397696 to 7316635045479571456. Current Snapshot: MvccSnapshot[applied={T|T < 7316635045802397696}]
I20260809 15:04:13.035884 16793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:13.036168 16793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:13.040186 16793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1569262e6342452a9740eee13ae17dca: Sending DeleteTablet for 3 replicas of tablet 1b21400d184340dea6adb60d921dc900
I20260809 15:04:13.041548 16991 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b21400d184340dea6adb60d921dc900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:13 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:59158
I20260809 15:04:13.042479 17247 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b21400d184340dea6adb60d921dc900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:13 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:52796
I20260809 15:04:13.043051 17463 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b21400d184340dea6adb60d921dc900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:13 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:53050
I20260809 15:04:13.051611 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 16846
I20260809 15:04:13.056514 17662 tablet_replica.cc:330] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: stopping tablet replica
I20260809 15:04:13.057512 17662 raft_consensus.cc:2229] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:04:13.059123 17662 raft_consensus.cc:2258] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:13.060483 17656 tablet_replica.cc:330] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: stopping tablet replica
I20260809 15:04:13.060861 17656 raft_consensus.cc:2229] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:13.061335 17656 raft_consensus.cc:2258] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:13.064443 17656 ts_tablet_manager.cc:1905] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:13.071211 17656 ts_tablet_manager.cc:1918] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:13.071415 17656 log.cc:1196] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/1b21400d184340dea6adb60d921dc900
I20260809 15:04:13.071830 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:13.052595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.27.186.129:38931)
0809 15:04:13.052784 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:13.052791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:13.052858 (+ 67us) server_negotiation.cc:407] Connection header received
0809 15:04:13.053045 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:13.053051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:13.053137 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:13.053242 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:13.055293 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:13.056450 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:13.058488 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:13.059201 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:13.059316 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:13.063119 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:13.063135 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:13.063153 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:13.063195 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:13.067315 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:13.068196 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:13.068205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:13.068208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:13.068388 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:13.068890 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:13.068896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:13.068899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:13.069386 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:13.070765 (+ 1379us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:13.070924 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 15:04:13.071150 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:04:13.073346 17103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:13.078529 17661 tablet_replica.cc:330] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: stopping tablet replica
I20260809 15:04:13.078948 17661 raft_consensus.cc:2229] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:13.071979 17656 ts_tablet_manager.cc:1939] T 1b21400d184340dea6adb60d921dc900 P 7949abd036f0499a9a4987217a1f52d0: Deleting consensus metadata
I20260809 15:04:13.079452 17661 raft_consensus.cc:2258] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:13.081453 17661 ts_tablet_manager.cc:1905] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:13.085228 17662 ts_tablet_manager.cc:1905] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:13.088099 17661 ts_tablet_manager.cc:1918] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:13.088294 17661 log.cc:1196] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/1b21400d184340dea6adb60d921dc900
I20260809 15:04:13.088845 17661 ts_tablet_manager.cc:1939] T 1b21400d184340dea6adb60d921dc900 P 341c1d2a78714b7fb04016f11336843f: Deleting consensus metadata
I20260809 15:04:13.090763 16768 catalog_manager.cc:4841] TS 341c1d2a78714b7fb04016f11336843f (127.27.186.130:45823): tablet 1b21400d184340dea6adb60d921dc900 (table test-table [id=cf5b63c6d99f4c02a3b2725184a5d40c]) successfully deleted
I20260809 15:04:13.095600 17662 ts_tablet_manager.cc:1918] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:13.095852 17662 log.cc:1196] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/1b21400d184340dea6adb60d921dc900
I20260809 15:04:13.096498 17662 ts_tablet_manager.cc:1939] T 1b21400d184340dea6adb60d921dc900 P 9f8c3a226f6a48d8be7b75a725cbda7d: Deleting consensus metadata
I20260809 15:04:13.098708 16766 catalog_manager.cc:4841] TS 9f8c3a226f6a48d8be7b75a725cbda7d (127.27.186.131:33627): tablet 1b21400d184340dea6adb60d921dc900 (table test-table [id=cf5b63c6d99f4c02a3b2725184a5d40c]) successfully deleted
I20260809 15:04:13.249336 16766 catalog_manager.cc:4841] TS 7949abd036f0499a9a4987217a1f52d0 (127.27.186.129:38931): tablet 1b21400d184340dea6adb60d921dc900 (table test-table [id=cf5b63c6d99f4c02a3b2725184a5d40c]) successfully deleted
I20260809 15:04:13.249431 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 16846
I20260809 15:04:13.278844 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 17117
I20260809 15:04:13.297251 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:13.279790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.27.186.130:45823)
0809 15:04:13.279979 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:13.279986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:13.280042 (+ 56us) server_negotiation.cc:407] Connection header received
0809 15:04:13.280279 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:13.280284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:13.280371 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:13.280490 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:13.282611 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:13.283772 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:13.285646 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:13.286277 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:13.286399 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:13.289730 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:13.289748 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:13.289764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:13.289807 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:13.293732 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:13.294687 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:13.294696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:13.294700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:13.294891 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:13.295375 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:13.295380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:13.295383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:13.295841 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:13.296216 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:13.296555 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 15:04:13.296787 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:04:13.297976 17343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:13.445814 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 17117
I20260809 15:04:13.474086 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 17355
I20260809 15:04:13.488623 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:13.474899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.27.186.131:33627)
0809 15:04:13.475692 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:13.475697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:13.475712 (+ 15us) server_negotiation.cc:407] Connection header received
0809 15:04:13.475771 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:13.475775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:13.475846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:13.475953 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:13.477331 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:13.478281 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:13.479557 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:13.479992 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:13.480104 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:13.482668 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:13.482681 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:13.482696 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:13.482738 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:13.485697 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:13.486471 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:13.486478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:13.486480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:13.486651 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:13.487012 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:13.487017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:13.487021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:13.487345 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:13.487628 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:13.488064 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 15:04:13.488272 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":587,"threads_started":1}
I20260809 15:04:13.489295 17574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:13.608217 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 17355
I20260809 15:04:13.632462 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 16739
I20260809 15:04:13.633577 16838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:13.757755 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 16739
I20260809 15:04:13.786073 28394 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 15:04:14.120105 28394 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 15:04:14.120312 28394 mini_ranger.cc:66] Stopping Ranger...
I20260809 15:04:14.463934 28394 mini_ranger.cc:68] Stopped Ranger
2026-08-09 15:04:14.464 UTC [15203] LOG: received smart shutdown request
2026-08-09 15:04:14.467 UTC [15203] LOG: background worker "logical replication launcher" (PID 15215) exited with exit code 1
2026-08-09 15:04:14.467 UTC [15210] LOG: shutting down
2026-08-09 15:04:14.602 UTC [15203] LOG: database system is shut down
2026-08-09T15:04:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50527 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:04:14 dist-test-slave-q9h9 krb5kdc[17690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:04:14 dist-test-slave-q9h9 krb5kdc[17690](info): set up 2 sockets
Aug 09 15:04:14 dist-test-slave-q9h9 krb5kdc[17690](info): commencing operation
krb5kdc: starting...
W20260809 15:04:15.912104 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:04:16 dist-test-slave-q9h9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:04:16.122987 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:46429
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:46429,127.27.186.189:46329,127.27.186.188:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:16.432962 17707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:16.433204 17707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:16.433327 17707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:16.440227 17707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:16.440315 17707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:16.440369 17707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:16.440444 17707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:16.440498 17707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:16.449644 17707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:46429,127.27.186.189:46329,127.27.186.188:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:16.450753 17707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:16.451896 17707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:16.462827 17719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:16.462990 17720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:16.463408 17707 server_base.cc:1034] running on GCE node
W20260809 15:04:16.463508 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:16.464216 17707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:16.465467 17707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:16.466681 17707 hybrid_clock.cc:648] HybridClock initialized: now 1786287856466629 us; error 44 us; skew 500 ppm
I20260809 15:04:16.466982 17707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:16 dist-test-slave-q9h9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:16.469789 17707 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:16.471143 17707 webserver.cc:460] Webserver started at http://127.27.186.190:43013/ using document root <none> and password file <none>
I20260809 15:04:16.471719 17707 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:16.471814 17707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:16.472038 17707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:16.474967 17707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "5d44710ed426456785359779c15f9d45"
format_stamp: "Formatted at 2026-08-09 15:04:16 on dist-test-slave-q9h9"
server_key: "3ffba583f22c70ccdf15d995c649e5b1"
server_key_iv: "05edb8e97eb9cb733b56684adc37392f"
server_key_version: "encryptionkey@0"
I20260809 15:04:16.475677 17707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "5d44710ed426456785359779c15f9d45"
format_stamp: "Formatted at 2026-08-09 15:04:16 on dist-test-slave-q9h9"
server_key: "3ffba583f22c70ccdf15d995c649e5b1"
server_key_iv: "05edb8e97eb9cb733b56684adc37392f"
server_key_version: "encryptionkey@0"
I20260809 15:04:16.480329 17707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:04:16.483491 17735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:16.484658 17707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 15:04:16.484817 17707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "5d44710ed426456785359779c15f9d45"
format_stamp: "Formatted at 2026-08-09 15:04:16 on dist-test-slave-q9h9"
server_key: "3ffba583f22c70ccdf15d995c649e5b1"
server_key_iv: "05edb8e97eb9cb733b56684adc37392f"
server_key_version: "encryptionkey@0"
I20260809 15:04:16.484984 17707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:16.500936 17707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:16.504513 17707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:16.504748 17707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:16.521382 17707 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:46429
I20260809 15:04:16.521422 17823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:46429 every 8 connection(s)
I20260809 15:04:16.522608 17707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:16.525878 17824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:16.532176 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 17707
I20260809 15:04:16.532415 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:04:16.531908 17824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:04:16.533244 28394 external_mini_cluster.cc:1428] Setting key 15d18fa9d8065ae6f53ff3bfec63cf9b
Aug 09 15:04:16 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:16.554212 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.535317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.27.186.190:46429)
0809 15:04:16.537030 (+ 1713us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:16.537039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:16.537087 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:04:16.538877 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:16.538883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:16.539084 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:16.539232 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:16.540599 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:16.541731 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:16.543480 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:16.544004 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:16.544144 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:16.547568 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:16.547576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:16.547586 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:16.547613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:16.550317 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:16.551142 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:16.551148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:16.551150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:16.551300 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:16.551727 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:16.551730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:16.551732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:16.552237 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:16.552387 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:16.552969 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 15:04:16.553263 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":1549,"thread_start_us":1249,"threads_started":1}
I20260809 15:04:16.554242 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.535317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46429 (local address 127.0.0.1:34408)
0809 15:04:16.536145 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.536200 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:16.538508 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:16.539313 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:16.539320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:16.539987 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:16.540413 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:16.540434 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:16.541899 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:16.541904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:16.543350 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:16.543364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:16.544239 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:16.544244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:16.544368 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:16.544946 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:16.544967 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:16.547356 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:16.550447 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:16.550459 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:16.550469 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:16.550982 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:16.551420 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:16.551424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:16.551427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:16.551622 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:16.552380 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:16.552387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:16.552610 (+ 223us) client_negotiation.cc:769] Sending connection context
0809 15:04:16.552854 (+ 244us) client_negotiation.cc:240] Negotiation successful
0809 15:04:16.553428 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":111,"thread_start_us":360,"threads_started":1}
I20260809 15:04:16.556916 17824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:04:16.557924 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.557596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46348)
0809 15:04:16.557739 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.557834 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 15:04:16.558189 17738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.186.189@KRBTEST.COM; defaulting to no policy
I20260809 15:04:16.559654 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.559334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46362)
0809 15:04:16.559485 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.559545 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 15:04:16.559854 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:46329: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab.
Entry for principal kudu/127.27.186.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab.
I20260809 15:04:16.587160 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:46329
--webserver_interface=127.27.186.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab
--principal=kudu/127.27.186.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:46429,127.27.186.189:46329,127.27.186.188:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:16.609206 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } attempt: 1
I20260809 15:04:16.610337 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.609833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46378)
0809 15:04:16.610067 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.610199 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 15:04:16.611416 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.611074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46380)
0809 15:04:16.611212 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.611288 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 15:04:16.611629 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:46329: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
I20260809 15:04:16.679857 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } attempt: 2
I20260809 15:04:16.680951 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.680510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46388)
0809 15:04:16.680726 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.680816 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 15:04:16.681963 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.681619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46390)
0809 15:04:16.681750 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.681832 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 15:04:16.682180 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:46329: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
I20260809 15:04:16.773420 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } attempt: 3
I20260809 15:04:16.774664 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.774218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46392)
0809 15:04:16.774414 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.774523 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 15:04:16.775650 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.775321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46398)
0809 15:04:16.775441 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.775507 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 15:04:16.775859 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:46329: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
W20260809 15:04:16.903381 17832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:16.903632 17832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:16.903724 17832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:16.911369 17832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:16.911469 17832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:16.911525 17832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:16.911580 17832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:16.911623 17832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:16.919078 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } attempt: 4
I20260809 15:04:16.920162 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.919648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46414)
0809 15:04:16.919941 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.920027 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260809 15:04:16.921192 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:16.920863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46418)
0809 15:04:16.920987 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:16.921078 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 15:04:16.921391 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:46329: Network error: Client connection negotiation failed: client connection to 127.27.186.189:46329: connect: Connection refused (error 111)
I20260809 15:04:16.921634 17832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:46429,127.27.186.189:46329,127.27.186.188:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab
--principal=kudu/127.27.186.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:46329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.186.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:16.922798 17832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:16.923930 17832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:16.934927 17841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:16.934921 17840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:16.935166 17843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:16.936131 17832 server_base.cc:1034] running on GCE node
I20260809 15:04:16.936839 17832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:16.938099 17832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:16.939308 17832 hybrid_clock.cc:648] HybridClock initialized: now 1786287856939251 us; error 48 us; skew 500 ppm
I20260809 15:04:16.939615 17832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:16 dist-test-slave-q9h9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:16.942631 17832 init.cc:377] Logged in from keytab as kudu/127.27.186.189@KRBTEST.COM (short username kudu)
I20260809 15:04:16.943955 17832 webserver.cc:460] Webserver started at http://127.27.186.189:42393/ using document root <none> and password file <none>
I20260809 15:04:16.944563 17832 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:16.944677 17832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:16.944962 17832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:16.947840 17832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data/instance:
uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1"
format_stamp: "Formatted at 2026-08-09 15:04:16 on dist-test-slave-q9h9"
server_key: "abe90c58b18552bcc1e68ba506ccc804"
server_key_iv: "53706b86cdd9a8a2143b42c78744e272"
server_key_version: "encryptionkey@0"
I20260809 15:04:16.948581 17832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal/instance:
uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1"
format_stamp: "Formatted at 2026-08-09 15:04:16 on dist-test-slave-q9h9"
server_key: "abe90c58b18552bcc1e68ba506ccc804"
server_key_iv: "53706b86cdd9a8a2143b42c78744e272"
server_key_version: "encryptionkey@0"
I20260809 15:04:16.953465 17832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 15:04:16.956835 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:16.958137 17832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:16.958314 17832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal
uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1"
format_stamp: "Formatted at 2026-08-09 15:04:16 on dist-test-slave-q9h9"
server_key: "abe90c58b18552bcc1e68ba506ccc804"
server_key_iv: "53706b86cdd9a8a2143b42c78744e272"
server_key_version: "encryptionkey@0"
I20260809 15:04:16.958488 17832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:16.990833 17832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:16.994524 17832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:16.994768 17832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:17.011821 17832 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.189:46329
I20260809 15:04:17.011878 17943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.189:46329 every 8 connection(s)
I20260809 15:04:17.013078 17832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
I20260809 15:04:17.016288 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 17832
I20260809 15:04:17.016454 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-1/wal/instance
I20260809 15:04:17.016683 17944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:17.016907 28394 external_mini_cluster.cc:1428] Setting key 81c326729baf7896ebcca18f2ce6e22e
I20260809 15:04:17.022856 17944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.186.188@KRBTEST.COM; defaulting to no policy
I20260809 15:04:17.044631 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.025373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.27.186.190:46429)
0809 15:04:17.025527 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.025532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.027083 (+ 1551us) server_negotiation.cc:407] Connection header received
0809 15:04:17.029610 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.029615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.029684 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.029780 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.031641 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.032309 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.034441 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.034884 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.034979 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.038666 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.038673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.038678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.038740 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.041643 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.042395 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.042399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.042401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.042523 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.042917 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.042921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.042923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.043204 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.043337 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.044035 (+ 698us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.044315 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:04:17.045413 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.026141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46429 (local address 127.0.0.1:34418)
0809 15:04:17.026899 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.026948 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.029308 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.030010 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.030042 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.030781 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.031466 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.031489 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.032469 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.032475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.034286 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.034301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.035100 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.035107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.035352 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.036014 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.036045 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.038432 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.041781 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.041788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.041798 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.042276 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.042630 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.042634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.042636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.042802 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.043332 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.043339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.043658 (+ 319us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.043891 (+ 233us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.044342 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":304,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.188@KRBTEST.COM" created.
I20260809 15:04:17.047087 17944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:04:17.061961 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.048122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.27.186.189:46329)
0809 15:04:17.048758 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.048765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.048793 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:04:17.048836 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.048842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.049017 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.049134 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.049969 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.051256 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.052691 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.053336 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.053435 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.056546 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.056557 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.056567 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.056598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.059246 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.059964 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.059970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.059972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.060097 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.060493 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.060496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.060498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.060827 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.060973 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.061351 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.061592 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":487,"thread_start_us":344,"threads_started":1}
I20260809 15:04:17.062216 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.047864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46422)
0809 15:04:17.048014 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.048029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.048115 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.049185 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.049190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.049614 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.049811 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.049822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.051410 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.051414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.052555 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.052570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.053476 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.053482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.053680 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.054242 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.054258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.056347 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.059384 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.059389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.059393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.059854 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.060225 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.060229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.060232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.060388 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.060955 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.060959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.061116 (+ 157us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.061264 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.061740 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":134}
I20260809 15:04:17.064376 17944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:04:17.065526 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.065024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38246)
0809 15:04:17.065230 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.065350 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 15:04:17.065816 17852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.067394 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.066996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38258)
0809 15:04:17.067172 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.067273 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 15:04:17.067621 17944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab.
Entry for principal kudu/127.27.186.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab.
I20260809 15:04:17.075398 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:45369
--webserver_interface=127.27.186.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab
--principal=kudu/127.27.186.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:46429,127.27.186.189:46329,127.27.186.188:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:17.116956 17944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 1
I20260809 15:04:17.118153 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.117610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38272)
0809 15:04:17.117814 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.117937 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 15:04:17.119120 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.118757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38276)
0809 15:04:17.118917 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.119002 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:04:17.119347 17944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.187599 17944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 2
I20260809 15:04:17.188882 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.188306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38290)
0809 15:04:17.188521 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.188656 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 15:04:17.190090 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.189642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38306)
0809 15:04:17.189807 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.189904 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 15:04:17.190363 17944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.220628 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } attempt: 5
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:04:17.234694 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.221298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46434)
0809 15:04:17.221508 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.221529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.221675 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.222069 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.222074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.222476 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.222665 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.222675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.223655 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.223659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.224528 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.224541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.225305 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.225310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.225500 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.226151 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.226169 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.228214 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.232277 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.232281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.232284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.232647 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.233100 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.233104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.233106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.233277 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.233907 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.233911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.234095 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.234246 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.234454 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 15:04:17.234953 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.221369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.27.186.189:46329)
0809 15:04:17.221561 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.221567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.221588 (+ 21us) server_negotiation.cc:407] Connection header received
0809 15:04:17.221828 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.221832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.221900 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.221986 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.222833 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.223518 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.224679 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.225163 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.225277 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.228428 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.228437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.228442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.228486 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.232109 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.232771 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.232777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.232781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.232959 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.233394 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.233399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.233402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.233767 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.233926 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.234397 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.234632 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 15:04:17.236055 17824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:04:17.237099 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.236733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38322)
0809 15:04:17.236891 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.236976 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 15:04:17.237995 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.237673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38334)
0809 15:04:17.237808 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.237890 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 15:04:17.238262 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.281620 17944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 3
I20260809 15:04:17.282815 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.282346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38350)
0809 15:04:17.282544 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.282634 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 15:04:17.284008 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.283560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38358)
0809 15:04:17.283803 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.283888 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260809 15:04:17.284247 17944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.289460 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 1
I20260809 15:04:17.290369 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.289932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38372)
0809 15:04:17.290131 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.290221 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 15:04:17.291270 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.290960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38384)
0809 15:04:17.291094 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.291155 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:04:17.291455 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.359651 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 2
I20260809 15:04:17.360684 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.360238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38400)
0809 15:04:17.360440 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.360522 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 15:04:17.361620 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.361275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38410)
0809 15:04:17.361439 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.361503 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 15:04:17.361883 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
W20260809 15:04:17.411379 17956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:17.411649 17956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:17.411789 17956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:17.418849 17956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:17.418954 17956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:17.419034 17956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:17.419113 17956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:17.419183 17956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:17.428575 17956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:46429,127.27.186.189:46329,127.27.186.188:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab
--principal=kudu/127.27.186.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.186.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:17.429773 17956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:17.431016 17956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 15:04:17.431229 17944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 4
I20260809 15:04:17.432430 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.431875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38422)
0809 15:04:17.432126 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.432235 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 15:04:17.433598 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.433194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38434)
0809 15:04:17.433356 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.433447 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 15:04:17.433887 17944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
W20260809 15:04:17.444617 17965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:17.445051 17968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:17.445215 17961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:17.451435 17956 server_base.cc:1034] running on GCE node
I20260809 15:04:17.452310 17956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:17.453977 17956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:17.455258 17956 hybrid_clock.cc:648] HybridClock initialized: now 1786287857455206 us; error 33 us; skew 500 ppm
I20260809 15:04:17.455650 17956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287857, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:17.459635 17956 init.cc:377] Logged in from keytab as kudu/127.27.186.188@KRBTEST.COM (short username kudu)
I20260809 15:04:17.461292 17956 webserver.cc:460] Webserver started at http://127.27.186.188:46465/ using document root <none> and password file <none>
I20260809 15:04:17.462172 17956 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:17.462318 17956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:17.462656 17956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:17.466683 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data/instance:
uuid: "42db1a79b75e49c48f04598f8fa0516f"
format_stamp: "Formatted at 2026-08-09 15:04:17 on dist-test-slave-q9h9"
server_key: "aac671d02c57cbff35a40d3919a8dd4d"
server_key_iv: "16316510b7d9da597858a064b12dc262"
server_key_version: "encryptionkey@0"
I20260809 15:04:17.467724 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal/instance:
uuid: "42db1a79b75e49c48f04598f8fa0516f"
format_stamp: "Formatted at 2026-08-09 15:04:17 on dist-test-slave-q9h9"
server_key: "aac671d02c57cbff35a40d3919a8dd4d"
server_key_iv: "16316510b7d9da597858a064b12dc262"
server_key_version: "encryptionkey@0"
I20260809 15:04:17.468086 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 3
I20260809 15:04:17.469008 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.468582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38436)
0809 15:04:17.468788 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.468867 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 15:04:17.470007 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.469643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38448)
0809 15:04:17.469844 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.469913 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260809 15:04:17.470288 17824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:45369: Network error: Client connection negotiation failed: client connection to 127.27.186.188:45369: connect: Connection refused (error 111)
I20260809 15:04:17.474184 17956 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 15:04:17.478400 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:17.479892 17956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 15:04:17.480067 17956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal
uuid: "42db1a79b75e49c48f04598f8fa0516f"
format_stamp: "Formatted at 2026-08-09 15:04:17 on dist-test-slave-q9h9"
server_key: "aac671d02c57cbff35a40d3919a8dd4d"
server_key_iv: "16316510b7d9da597858a064b12dc262"
server_key_version: "encryptionkey@0"
I20260809 15:04:17.480293 17956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:17.506582 17956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:17.511560 17956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:17.511849 17956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:17.532030 18058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.188:45369 every 8 connection(s)
I20260809 15:04:17.532027 17956 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.188:45369
I20260809 15:04:17.533664 17956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
I20260809 15:04:17.536209 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 17956
I20260809 15:04:17.536404 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786287660051758-28394-0/minicluster-data/master-2/wal/instance
I20260809 15:04:17.536948 28394 external_mini_cluster.cc:1428] Setting key 80ec5bfa067de1d51f8e27133382f767
I20260809 15:04:17.537662 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:17.547372 18059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287857, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:04:17.568786 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.550925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.27.186.190:46429)
0809 15:04:17.551444 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.551450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.551825 (+ 375us) server_negotiation.cc:407] Connection header received
0809 15:04:17.553300 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.553305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.553387 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.553503 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.555359 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.556454 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.558764 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.559350 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.559460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.562584 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.562592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.562597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.562639 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.565424 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.566341 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.566347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.566351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.566484 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.566929 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.566932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.566934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.567221 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.567375 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.568108 (+ 733us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.568345 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":778,"threads_started":1}
I20260809 15:04:17.570007 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.550471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:46429 (local address 127.0.0.1:34434)
0809 15:04:17.551381 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.551420 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.553022 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.553725 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.553732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.554587 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.555138 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.555160 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.556635 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.556641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.558601 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.558617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.559577 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.559586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.559718 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.560256 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.560277 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.562375 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.565578 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.565590 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.565605 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.566206 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.566596 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.566599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.566601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.566805 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.567359 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.567366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.567669 (+ 303us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.567972 (+ 303us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.568401 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":53,"thread_start_us":569,"threads_started":1}
I20260809 15:04:17.570730 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.538868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.27.186.188:45369)
0809 15:04:17.551244 (+ 12376us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.551257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.551302 (+ 45us) server_negotiation.cc:407] Connection header received
0809 15:04:17.553006 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.553034 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.553334 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.553497 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.555068 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.556767 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.558237 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.558854 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.559021 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.563721 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.563733 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.563747 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.563790 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.567484 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.568357 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.568364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.568368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.568526 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.568972 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.568977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.568979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.569395 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.569681 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.570101 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.570330 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":12142,"thread_start_us":11897,"threads_started":1}
I20260809 15:04:17.572275 18059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287857, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:17.586725 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.573352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.27.186.189:46329)
0809 15:04:17.573525 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.573530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.573549 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:04:17.573605 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.573610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.573685 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.573775 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.574725 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.575800 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.577060 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.577519 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.577628 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.580473 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.580478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.580481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.580513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.583622 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.584363 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.584369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.584372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.584549 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.585108 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.585113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.585117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.585493 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.585646 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.586114 (+ 468us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.586334 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:04:17.586740 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.573007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:46329 (local address 127.0.0.1:46448)
0809 15:04:17.573163 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.573181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.573269 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.573926 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.573929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.574360 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.574547 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.574558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.575958 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.575963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.576900 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.576912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.577656 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.577662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.577836 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.578383 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.578404 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.580281 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.583804 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.583809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.583814 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.584224 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.584705 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.584709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.584725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.584894 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.585630 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.585635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.585797 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.585956 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.586393 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 15:04:17.588137 18059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:04:17.590914 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.573658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.27.186.190:46429)
0809 15:04:17.573808 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.573813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.573830 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:04:17.573878 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.573883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.573964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.574095 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.575418 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.576132 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.577956 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.578489 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.578612 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.583481 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.583488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.583493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.583530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.587102 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.587828 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.587833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.587837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.587987 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.588781 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.588786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.588790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.589143 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.590199 (+ 1056us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.590365 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.590588 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287857, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:04:17.602730 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.589177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.27.186.188:45369)
0809 15:04:17.589369 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.589374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.589392 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:04:17.589436 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.589441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.589503 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.589594 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.590387 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.591028 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.592388 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.593022 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.593151 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.596816 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.596823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.596828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.596864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.599926 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.600705 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.600725 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.600729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.600893 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.601314 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.601319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.601322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.601666 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.601822 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.602233 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.602439 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 15:04:17.603525 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.588838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38456)
0809 15:04:17.589039 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.589057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.589148 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.589646 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.589650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.590100 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.590274 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.590285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.591158 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.591163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.592240 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.592256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.593321 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.593327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.593528 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.594125 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.594146 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.596625 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.600070 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.600075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.600079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.600563 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.601026 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.601031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.601035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.601193 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.601800 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.601805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.601971 (+ 166us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.602925 (+ 954us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.603198 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 15:04:17.608042 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.592740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.27.186.189:46329)
0809 15:04:17.592878 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.592881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.592893 (+ 12us) server_negotiation.cc:407] Connection header received
0809 15:04:17.592927 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.592930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.592977 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.593053 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.594319 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.595050 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.596543 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.597118 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.597230 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.602184 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.602189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.602194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.602225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.605205 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.606034 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.606040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.606043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.606243 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.606631 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.606636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.606639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.606978 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.607256 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.607528 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.607738 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:04:17.609048 18059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: Bootstrap starting.
I20260809 15:04:17.612634 18059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:17.613168 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:04:17.613664 18059 log.cc:824] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:17.616103 18059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: No bootstrap required, opened a new log
I20260809 15:04:17.623240 18059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.623517 18059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:17.623600 18059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db1a79b75e49c48f04598f8fa0516f, State: Initialized, Role: FOLLOWER
I20260809 15:04:17.624286 18059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.625888 18081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.626458 18081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.627326 18059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:17.647477 17824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 4
I20260809 15:04:17.656299 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.637105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.27.186.190:46429)
0809 15:04:17.637323 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.637331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.637358 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:04:17.637436 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.637443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.637533 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.637680 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.640377 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.641566 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.643750 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.644394 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.644520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.648574 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.648581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.648586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.648653 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.652173 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.653193 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.653199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.653203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.653352 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.653856 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.653861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.653865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.654270 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.654509 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.655697 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.655921 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 15:04:17.664288 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.624545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.27.186.189:46329)
0809 15:04:17.624724 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.624730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.638062 (+ 13332us) server_negotiation.cc:407] Connection header received
0809 15:04:17.638138 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.638143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.638226 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.638322 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.641256 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.642388 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.644762 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.645423 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.645525 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.655360 (+ 9835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.655367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.655372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.655413 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.659564 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.660555 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.660561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.660565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.660772 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.662088 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.662093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.662097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.662487 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.662812 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.663679 (+ 867us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.663928 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:04:17.675855 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.637020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.27.186.188:45369)
0809 15:04:17.638576 (+ 1556us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.638582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.655505 (+ 16923us) server_negotiation.cc:407] Connection header received
0809 15:04:17.658385 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.658391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.658470 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.658537 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.660063 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.661196 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.663881 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.664593 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.664704 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.668386 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.668392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.668397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.668434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.671989 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.672807 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.672813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.672816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.673034 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.673517 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.673523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.673526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.673971 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.674267 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.675236 (+ 969us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.675458 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":68}
W20260809 15:04:17.682852 18098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:04:17.682979 18098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:04:17.683501 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.648062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38470)
0809 15:04:17.648308 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.648327 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.648424 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.666458 (+ 18034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.666464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.666913 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.667132 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.667146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.669622 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.669629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.671234 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.671249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.672205 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.672221 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.672397 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.672971 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.672994 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.675777 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.680572 (+ 4795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.680577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.680581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.681087 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.681595 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.681600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.681603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.681786 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.682560 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.682565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.682740 (+ 175us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.682887 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.683150 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260809 15:04:17.684116 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.648695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.27.186.188:45369)
0809 15:04:17.666158 (+ 17463us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.666164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.666186 (+ 22us) server_negotiation.cc:407] Connection header received
0809 15:04:17.666231 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.666236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.666307 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.666408 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.667352 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.669433 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.671408 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.672036 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.673709 (+ 1673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.676274 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.676281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.676286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.676327 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.680393 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.681251 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.681255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.681258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.681437 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.681951 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.681955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.681958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.682393 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.683222 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.683345 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.683714 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":17290,"thread_start_us":17140,"threads_started":1}
I20260809 15:04:17.674853 17956 master_runner.cc:186] Error getting master registration for 127.27.186.190:46429: OK, instance_id { permanent_uuid: "5d44710ed426456785359779c15f9d45" instance_seqno: 1786287856516664 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 15:04:17.686069 17956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 15:04:17.690919 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: Bootstrap starting.
I20260809 15:04:17.695412 17824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:17.696431 17824 log.cc:824] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:17.699002 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: No bootstrap required, opened a new log
I20260809 15:04:17.706347 17824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.706621 17824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:17.706733 17824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d44710ed426456785359779c15f9d45, State: Initialized, Role: FOLLOWER
I20260809 15:04:17.707258 17824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.708113 18104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.708536 18104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.709369 17824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 15:04:17.718371 18123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:04:17.718502 18123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:04:17.717973 17707 master_runner.cc:186] Error getting master registration for 127.27.186.189:46329: OK, instance_id { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" instance_seqno: 1786287857007025 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 15:04:17.729889 17707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 15:04:17.733104 17944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } attempt: 5
I20260809 15:04:17.736554 18081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:17.736837 18081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.738399 18081 leader_election.cc:288] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d44710ed426456785359779c15f9d45 (127.27.186.190:46429), de4913cce7b3415d8d5a1ad91d1dd5e1 (127.27.186.189:46329)
I20260809 15:04:17.740134 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42db1a79b75e49c48f04598f8fa0516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d44710ed426456785359779c15f9d45" is_pre_election: true
I20260809 15:04:17.740731 17780 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db1a79b75e49c48f04598f8fa0516f in term 0.
I20260809 15:04:17.741417 17980 leader_election.cc:302] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db1a79b75e49c48f04598f8fa0516f, 5d44710ed426456785359779c15f9d45; no voters:
Aug 09 15:04:17 dist-test-slave-q9h9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287856, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:04:17.744742 18081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:04:17.744918 18081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:04:17.745012 18081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:17.748668 18081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.751304 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42db1a79b75e49c48f04598f8fa0516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d44710ed426456785359779c15f9d45"
I20260809 15:04:17.751526 17780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:17.754769 17780 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db1a79b75e49c48f04598f8fa0516f in term 1.
I20260809 15:04:17.754973 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.733630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:45369 (local address 127.0.0.1:38486)
0809 15:04:17.733984 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:17.734003 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:17.734141 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:17.734503 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:17.734508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:17.735047 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:17.735261 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.735275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.736838 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.736844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.738404 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:17.738427 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.739700 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.739706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:17.741237 (+ 1531us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.741907 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:17.741932 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:17.745278 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:17.751316 (+ 6038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.751321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.751325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.751983 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.752739 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:17.752744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:17.752748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:17.752920 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:17.753914 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:17.753920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:17.754130 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 15:04:17.754263 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 15:04:17.754526 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260809 15:04:17.755199 17980 leader_election.cc:302] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db1a79b75e49c48f04598f8fa0516f, 5d44710ed426456785359779c15f9d45; no voters:
I20260809 15:04:17.757009 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:17.733968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.27.186.188:45369)
0809 15:04:17.734209 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:17.734215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:17.734234 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:04:17.734285 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:17.734290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:17.734360 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:17.734473 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:17.735424 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.736614 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.738660 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:17.739362 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:17.739497 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:17.745511 (+ 6014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:17.745518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:17.745523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:17.745562 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:17.751093 (+ 5531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.752230 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.752235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.752238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.752559 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:17.753108 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:17.753121 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:17.753125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:17.753708 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:17.755751 (+ 2043us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:17.756027 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 15:04:17.756426 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260809 15:04:17.757337 17745 tablet.cc:2369] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:17.757634 18081 leader_election.cc:288] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [CANDIDATE]: Term 1 election: Requested vote from peers 5d44710ed426456785359779c15f9d45 (127.27.186.190:46429), de4913cce7b3415d8d5a1ad91d1dd5e1 (127.27.186.189:46329)
I20260809 15:04:17.758409 18081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:17.758993 18081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [term 1 LEADER]: Becoming Leader. State: Replica: 42db1a79b75e49c48f04598f8fa0516f, State: Running, Role: LEADER
I20260809 15:04:17.761042 18081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.763031 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: Bootstrap starting.
W20260809 15:04:17.764117 17987 tablet.cc:2369] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:17.771148 17944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:17.773192 17944 log.cc:824] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:17.778827 18126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42db1a79b75e49c48f04598f8fa0516f. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.779287 18126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:17.785848 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: No bootstrap required, opened a new log
I20260809 15:04:17.786305 18129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:17.788998 18129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:17.806795 17780 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [term 1 FOLLOWER]: Refusing update from remote peer 42db1a79b75e49c48f04598f8fa0516f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:17.808522 18126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:17.808744 17944 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.809399 17944 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:17.809607 17944 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4913cce7b3415d8d5a1ad91d1dd5e1, State: Initialized, Role: FOLLOWER
I20260809 15:04:17.811076 17944 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } }
I20260809 15:04:17.823719 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42db1a79b75e49c48f04598f8fa0516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1"
I20260809 15:04:17.823755 17896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42db1a79b75e49c48f04598f8fa0516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" is_pre_election: true
I20260809 15:04:17.824028 17893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:17.825369 17944 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:17.827616 17893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [term 1 FOLLOWER]: Refusing update from remote peer 42db1a79b75e49c48f04598f8fa0516f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:17.828366 18126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:17.829382 18132 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.829809 18132 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.853196 18104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42db1a79b75e49c48f04598f8fa0516f. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.853514 18104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.856953 18131 mvcc.cc:205] Tried to move back new op lower bound from 7316635065561448448 to 7316635065414639616. Current Snapshot: MvccSnapshot[applied={T|T < 7316635065561448448}]
I20260809 15:04:17.860548 18133 mvcc.cc:205] Tried to move back new op lower bound from 7316635065561448448 to 7316635065414639616. Current Snapshot: MvccSnapshot[applied={T|T < 7316635065561448448}]
I20260809 15:04:17.863269 18132 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42db1a79b75e49c48f04598f8fa0516f. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.863569 18132 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.869392 18126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.869627 18126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:17.869956 18126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.870193 18126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:17.871954 18140 mvcc.cc:205] Tried to move back new op lower bound from 7316635065561448448 to 7316635065414639616. Current Snapshot: MvccSnapshot[applied={T|T < 7316635065561448448}]
I20260809 15:04:17.878357 18132 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.878588 18132 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.879295 18129 catalog_manager.cc:1347] Generated new cluster ID: 0ff771ec8177477c8cadc4b95e3ab88e
I20260809 15:04:17.879393 18129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:17.892540 18104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42db1a79b75e49c48f04598f8fa0516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d44710ed426456785359779c15f9d45" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 46429 } } peers { permanent_uuid: "de4913cce7b3415d8d5a1ad91d1dd5e1" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 46329 } } peers { permanent_uuid: "42db1a79b75e49c48f04598f8fa0516f" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 45369 } } }
I20260809 15:04:17.892808 18104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:04:17.894236 18129 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:04:17.903479 18129 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:17.934404 18129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42db1a79b75e49c48f04598f8fa0516f: Generated new TSK 0
I20260809 15:04:17.935523 18129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:04:17.950845 18163 catalog_manager.cc:1259] Loaded cluster ID: 0ff771ec8177477c8cadc4b95e3ab88e
I20260809 15:04:17.950950 18163 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: loading cluster ID for follower catalog manager: success
I20260809 15:04:17.959724 18163 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: acquiring CA information for follower catalog manager: success
I20260809 15:04:17.962110 18163 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de4913cce7b3415d8d5a1ad91d1dd5e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 15:04:18.720357 18123 catalog_manager.cc:1259] Loaded cluster ID: 0ff771ec8177477c8cadc4b95e3ab88e
I20260809 15:04:18.720458 18123 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: loading cluster ID for follower catalog manager: success
I20260809 15:04:18.722875 18123 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: acquiring CA information for follower catalog manager: success
I20260809 15:04:18.724561 18123 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d44710ed426456785359779c15f9d45: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 15:04:19.080359 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 17707
I20260809 15:04:19.081687 17818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:19.186615 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 17707
I20260809 15:04:19.215323 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 17832
I20260809 15:04:19.216357 17937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:19.321326 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 17832
W20260809 15:04:19.345566 17981 connection.cc:535] client connection to 127.27.186.189:46329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 15:04:19.345916 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 17956
W20260809 15:04:19.345965 17981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 15:04:19.347095 18051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:19.452405 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 17956
2026-08-09T15:04:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4751 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:04:19 dist-test-slave-q9h9 krb5kdc[18190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:04:19 dist-test-slave-q9h9 krb5kdc[18190](info): set up 2 sockets
Aug 09 15:04:19 dist-test-slave-q9h9 krb5kdc[18190](info): commencing operation
krb5kdc: starting...
W20260809 15:04:20.667979 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:04:20 dist-test-slave-q9h9 krb5kdc[18190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:04:20.881690 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:35687
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:21.195781 18204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:21.196044 18204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:21.196183 18204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:21.203004 18204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:21.203109 18204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:21.203191 18204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:21.203266 18204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:21.203326 18204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:21.212472 18204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:21.213603 18204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:21.214782 18204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:21.225555 18214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:21.225616 18220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:21.225594 18216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:21.226377 18204 server_base.cc:1034] running on GCE node
I20260809 15:04:21.227023 18204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:21.228235 18204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:21.229420 18204 hybrid_clock.cc:648] HybridClock initialized: now 1786287861229372 us; error 45 us; skew 500 ppm
I20260809 15:04:21.229722 18204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:21 dist-test-slave-q9h9 krb5kdc[18190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287861, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:21.237237 18204 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:21.238545 18204 webserver.cc:460] Webserver started at http://127.27.186.190:37427/ using document root <none> and password file <none>
I20260809 15:04:21.239114 18204 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:21.239218 18204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:21.239452 18204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:21.242159 18204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "f11b67e9c97f462888ae956626752c13"
format_stamp: "Formatted at 2026-08-09 15:04:21 on dist-test-slave-q9h9"
server_key: "779b8fe2342201c1c8b509d53a9227ef"
server_key_iv: "5e6110e428728d5466cfcf5085cee09e"
server_key_version: "encryptionkey@0"
I20260809 15:04:21.243109 18204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "f11b67e9c97f462888ae956626752c13"
format_stamp: "Formatted at 2026-08-09 15:04:21 on dist-test-slave-q9h9"
server_key: "779b8fe2342201c1c8b509d53a9227ef"
server_key_iv: "5e6110e428728d5466cfcf5085cee09e"
server_key_version: "encryptionkey@0"
I20260809 15:04:21.248250 18204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 15:04:21.252033 18236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:21.253312 18204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:04:21.253481 18204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "f11b67e9c97f462888ae956626752c13"
format_stamp: "Formatted at 2026-08-09 15:04:21 on dist-test-slave-q9h9"
server_key: "779b8fe2342201c1c8b509d53a9227ef"
server_key_iv: "5e6110e428728d5466cfcf5085cee09e"
server_key_version: "encryptionkey@0"
I20260809 15:04:21.253659 18204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:21.265545 18204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:21.269025 18204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:21.269265 18204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:21.286065 18204 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:35687
I20260809 15:04:21.286146 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:35687 every 8 connection(s)
I20260809 15:04:21.287307 18204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:21.289206 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 18204
I20260809 15:04:21.289423 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:04:21.290091 28394 external_mini_cluster.cc:1428] Setting key 5db1a5c81e082bebe29f23ff10b80dc5
I20260809 15:04:21.291055 18319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 15:04:21 dist-test-slave-q9h9 krb5kdc[18190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:21.304430 18319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13: Bootstrap starting.
I20260809 15:04:21.308207 18319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:21.309355 18319 log.cc:824] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:21.312435 18319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13: No bootstrap required, opened a new log
I20260809 15:04:21.312783 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:21.291761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.27.186.190:35687)
0809 15:04:21.292457 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:21.292468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:21.292506 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:04:21.294055 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:21.294080 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:21.294435 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:21.294733 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:21.296571 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:21.297862 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:21.299714 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:21.300339 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:21.300475 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:21.305134 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:21.305145 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:21.305162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:21.305211 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:21.308852 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:21.309489 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:21.309498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:21.309505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:21.309697 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:21.310142 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:21.310147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:21.310150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:21.310824 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:21.311031 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:21.311507 (+ 476us) server_negotiation.cc:299] Negotiation successful
0809 15:04:21.311809 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":310,"threads_started":1}
I20260809 15:04:21.316322 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:04:21.320864 18319 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b67e9c97f462888ae956626752c13" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35687 } }
I20260809 15:04:21.321169 18319 raft_consensus.cc:374] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:21.321242 18319 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11b67e9c97f462888ae956626752c13, State: Initialized, Role: FOLLOWER
I20260809 15:04:21.321681 18319 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b67e9c97f462888ae956626752c13" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35687 } }
I20260809 15:04:21.321869 18319 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:21.321959 18319 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:21.322155 18319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:21.324085 18319 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b67e9c97f462888ae956626752c13" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35687 } }
I20260809 15:04:21.324489 18319 leader_election.cc:302] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11b67e9c97f462888ae956626752c13; no voters:
I20260809 15:04:21.324972 18319 leader_election.cc:288] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:04:21.325085 18329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:21.325686 18329 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [term 1 LEADER]: Becoming Leader. State: Replica: f11b67e9c97f462888ae956626752c13, State: Running, Role: LEADER
I20260809 15:04:21.326197 18329 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b67e9c97f462888ae956626752c13" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35687 } }
I20260809 15:04:21.326828 18319 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:21.328821 18333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11b67e9c97f462888ae956626752c13. Latest consensus state: current_term: 1 leader_uuid: "f11b67e9c97f462888ae956626752c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b67e9c97f462888ae956626752c13" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35687 } } }
I20260809 15:04:21.329186 18333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:21.329147 18331 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f11b67e9c97f462888ae956626752c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b67e9c97f462888ae956626752c13" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35687 } } }
I20260809 15:04:21.329483 18331 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13 [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:21.331286 18343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:21.336265 18343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:21.344244 18343 catalog_manager.cc:1347] Generated new cluster ID: bc9dbdda94064b08af9b97ed6695a6b1
I20260809 15:04:21.344339 18343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:21.369858 18343 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:04:21.371302 18343 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:21.378458 18343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f11b67e9c97f462888ae956626752c13: Generated new TSK 0
I20260809 15:04:21.379351 18343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:04:21.428311 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 18204
I20260809 15:04:21.429399 18308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:21.521286 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 18204
2026-08-09T15:04:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2060 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:04:21 dist-test-slave-q9h9 krb5kdc[18361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:04:21 dist-test-slave-q9h9 krb5kdc[18361](info): set up 2 sockets
Aug 09 15:04:21 dist-test-slave-q9h9 krb5kdc[18361](info): commencing operation
krb5kdc: starting...
W20260809 15:04:22.723834 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:04:22 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:04:22.935456 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40839
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:23.241333 18386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:23.241586 18386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:23.241714 18386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:23.248523 18386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:23.248632 18386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:23.248677 18386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:23.248734 18386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:23.248778 18386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:23.258314 18386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:23.259418 18386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:23.260593 18386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:23.271180 18398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:23.271618 18393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:23.271663 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:23.272049 18386 server_base.cc:1034] running on GCE node
I20260809 15:04:23.272753 18386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:23.273974 18386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:23.275200 18386 hybrid_clock.cc:648] HybridClock initialized: now 1786287863275150 us; error 41 us; skew 500 ppm
I20260809 15:04:23.275494 18386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:23 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:23.278335 18386 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:23.279563 18386 webserver.cc:460] Webserver started at http://127.27.186.190:45309/ using document root <none> and password file <none>
I20260809 15:04:23.280153 18386 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:23.280249 18386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:23.280481 18386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:23.283278 18386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "51cf2597223245ff8a975aa72b946ccd"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "1bd4c417305110d654d0f1a28328d41b"
server_key_iv: "b4af4ebe527dcb77b30580891c6e2c73"
server_key_version: "encryptionkey@0"
I20260809 15:04:23.283927 18386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "51cf2597223245ff8a975aa72b946ccd"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "1bd4c417305110d654d0f1a28328d41b"
server_key_iv: "b4af4ebe527dcb77b30580891c6e2c73"
server_key_version: "encryptionkey@0"
I20260809 15:04:23.288537 18386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:04:23.291654 18408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:23.292819 18386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:04:23.292963 18386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "51cf2597223245ff8a975aa72b946ccd"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "1bd4c417305110d654d0f1a28328d41b"
server_key_iv: "b4af4ebe527dcb77b30580891c6e2c73"
server_key_version: "encryptionkey@0"
I20260809 15:04:23.293136 18386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:23.316625 18386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:23.320017 18386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:23.320252 18386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:23.336494 18386 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:40839
I20260809 15:04:23.336539 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:40839 every 8 connection(s)
I20260809 15:04:23.337632 18386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:23.340832 18492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:23.343428 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 18386
I20260809 15:04:23.343636 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:04:23.344435 28394 external_mini_cluster.cc:1428] Setting key 31feee3d1a7b3afc7efadb88a902fe31
I20260809 15:04:23.352334 18492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap starting.
I20260809 15:04:23.355806 18492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:23.357025 18492 log.cc:824] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:04:23 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:23.359669 18492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: No bootstrap required, opened a new log
I20260809 15:04:23.366143 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:23.346381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.27.186.190:40839)
0809 15:04:23.347070 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:23.347080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:23.347118 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:04:23.348662 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:23.348683 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:23.349031 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:23.349339 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:23.351046 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:23.352283 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:23.354134 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:23.354776 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:23.354917 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:23.359393 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:23.359400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:23.359413 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:23.359449 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:23.362435 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:23.363089 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:23.363095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:23.363100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:23.363252 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:23.363631 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:23.363635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:23.363637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:23.364120 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:23.364432 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:23.364855 (+ 423us) server_negotiation.cc:299] Negotiation successful
0809 15:04:23.365202 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":324,"threads_started":1}
I20260809 15:04:23.367671 18492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:23.368000 18492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:23.368090 18492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cf2597223245ff8a975aa72b946ccd, State: Initialized, Role: FOLLOWER
I20260809 15:04:23.368590 18492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:23.368850 18492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:23.368991 18492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:23.369143 18492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:23.371172 18492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:23.371598 18492 leader_election.cc:302] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cf2597223245ff8a975aa72b946ccd; no voters:
I20260809 15:04:23.372081 18492 leader_election.cc:288] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:04:23.372203 18499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:23.372824 18499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 LEADER]: Becoming Leader. State: Replica: 51cf2597223245ff8a975aa72b946ccd, State: Running, Role: LEADER
I20260809 15:04:23.373406 18499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:23.374058 18492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:23.376336 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cf2597223245ff8a975aa72b946ccd. Latest consensus state: current_term: 1 leader_uuid: "51cf2597223245ff8a975aa72b946ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } } }
I20260809 15:04:23.376313 18500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51cf2597223245ff8a975aa72b946ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } } }
I20260809 15:04:23.377144 18500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:23.377029 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:23.379060 18512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:23.384150 18512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:23.392849 18512 catalog_manager.cc:1347] Generated new cluster ID: 4f4712e8f2ce475393f3682c5e4bcd9a
I20260809 15:04:23.392943 18512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:23.416587 18512 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:04:23.417738 18512 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:23.427532 18512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Generated new TSK 0
I20260809 15:04:23.428246 18512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:04:23.492878 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:40839
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:23.824635 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:23.824898 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:23.825006 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:23.831857 18528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:23.831947 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:23.832057 18528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:04:23.841053 18528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:23.842062 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:23.843214 18528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:23.855221 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:23.855341 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:23.855309 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:23.856041 18528 server_base.cc:1034] running on GCE node
I20260809 15:04:23.856649 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:23.857472 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:23.858659 18528 hybrid_clock.cc:648] HybridClock initialized: now 1786287863858614 us; error 34 us; skew 500 ppm
I20260809 15:04:23.858943 18528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:23 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:23.861852 18528 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:04:23.863093 18528 webserver.cc:460] Webserver started at http://127.27.186.129:39857/ using document root <none> and password file <none>
I20260809 15:04:23.863668 18528 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:23.863766 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:23.863996 18528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:23.866791 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "921308cf39864144b8bed4446d8cf36a"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "0399b6f9db88c79851bf355b9111877d"
server_key_iv: "b0dbc7e1a8a32db9224a29e06c2e72b3"
server_key_version: "encryptionkey@0"
I20260809 15:04:23.867527 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "921308cf39864144b8bed4446d8cf36a"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "0399b6f9db88c79851bf355b9111877d"
server_key_iv: "b0dbc7e1a8a32db9224a29e06c2e72b3"
server_key_version: "encryptionkey@0"
I20260809 15:04:23.872279 18528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:04:23.875669 18550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:23.876837 18528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 15:04:23.876972 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "921308cf39864144b8bed4446d8cf36a"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "0399b6f9db88c79851bf355b9111877d"
server_key_iv: "b0dbc7e1a8a32db9224a29e06c2e72b3"
server_key_version: "encryptionkey@0"
I20260809 15:04:23.877156 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:23.897099 18528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:23.900547 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:23.900780 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:23.901552 18528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:23.902801 18528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:23.902872 18528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:23.902961 18528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:23.903002 18528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:23.931562 18528 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:45865
I20260809 15:04:23.931618 18720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:45865 every 8 connection(s)
I20260809 15:04:23.932658 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:04:23.936661 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 18528
I20260809 15:04:23.936844 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:04:23.937505 28394 external_mini_cluster.cc:1428] Setting key 29b39cd3f1a2edb27b951f71bb3bad57
Aug 09 15:04:23 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:23.951076 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:23.934953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:54943 (local address 127.27.186.190:40839)
0809 15:04:23.935374 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:23.935379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:23.936423 (+ 1044us) server_negotiation.cc:407] Connection header received
0809 15:04:23.938354 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:23.938359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:23.938438 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:23.938533 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:23.940040 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:23.940752 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:23.942264 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:23.942669 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:23.942790 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:23.945921 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:23.945926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:23.945929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:23.945959 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:23.948263 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:23.948914 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:23.948920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:23.948923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:23.949037 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:23.949434 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:23.949439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:23.949443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:23.949688 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:23.949830 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:23.950503 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 15:04:23.950745 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260809 15:04:23.951825 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:23.935461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:54943)
0809 15:04:23.936267 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:23.936303 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:23.938076 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:23.938773 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:23.938781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:23.939369 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:23.939880 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:23.939897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:23.940904 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:23.940908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:23.942125 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:23.942138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:23.942896 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:23.942903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:23.943117 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:23.943702 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:23.943729 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:23.945725 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:23.948388 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:23.948397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:23.948411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:23.948802 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:23.949138 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:23.949143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:23.949147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:23.949328 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:23.949817 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:23.949824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:23.950141 (+ 317us) client_negotiation.cc:769] Sending connection context
0809 15:04:23.950378 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 15:04:23.950830 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":361,"threads_started":1}
I20260809 15:04:23.953267 18722 heartbeater.cc:347] Connected to a master server at 127.27.186.190:40839
I20260809 15:04:23.953518 18722 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:23.954221 18722 heartbeater.cc:511] Master 127.27.186.190:40839 requested a full tablet report, sending...
I20260809 15:04:23.956393 18436 ts_manager.cc:194] Registered new tserver with Master: 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865)
I20260809 15:04:23.958261 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:54943
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:04:23.990954 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:40839
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:24.302095 18732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:24.302342 18732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:24.302475 18732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:24.309351 18732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:24.309446 18732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:24.309589 18732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:04:24.318769 18732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:24.319811 18732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:24.320952 18732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:24.333034 18739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:24.333077 18741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:24.333221 18744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:24.334040 18732 server_base.cc:1034] running on GCE node
I20260809 15:04:24.334558 18732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:24.335359 18732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:24.336547 18732 hybrid_clock.cc:648] HybridClock initialized: now 1786287864336493 us; error 46 us; skew 500 ppm
I20260809 15:04:24.336824 18732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:24 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:24.339855 18732 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:04:24.341200 18732 webserver.cc:460] Webserver started at http://127.27.186.130:46385/ using document root <none> and password file <none>
I20260809 15:04:24.341784 18732 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:24.341898 18732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:24.342211 18732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:24.344799 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "f35b6bc5b76144929d9f0fc6cc585393"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "ca0522b22d19a5f433eeea6e48b4774d"
server_key_iv: "817ec646be44412c81fed928cdc0050e"
server_key_version: "encryptionkey@0"
I20260809 15:04:24.345517 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "f35b6bc5b76144929d9f0fc6cc585393"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "ca0522b22d19a5f433eeea6e48b4774d"
server_key_iv: "817ec646be44412c81fed928cdc0050e"
server_key_version: "encryptionkey@0"
I20260809 15:04:24.350427 18732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260809 15:04:24.353535 18754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:24.354740 18732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:04:24.354890 18732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "f35b6bc5b76144929d9f0fc6cc585393"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "ca0522b22d19a5f433eeea6e48b4774d"
server_key_iv: "817ec646be44412c81fed928cdc0050e"
server_key_version: "encryptionkey@0"
I20260809 15:04:24.355057 18732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:24.371301 18732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:24.375119 18732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:24.375385 18732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:24.376214 18732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:24.377460 18732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:24.377534 18732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:24.377660 18732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:24.377714 18732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:24.407526 18732 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33789
I20260809 15:04:24.407569 18938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33789 every 8 connection(s)
I20260809 15:04:24.408696 18732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:04:24.409188 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 18732
I20260809 15:04:24.409349 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:04:24.409927 28394 external_mini_cluster.cc:1428] Setting key e02f089807338fde19c4c044629e5d67
Aug 09 15:04:24 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:24.427950 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.411933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:59377 (local address 127.27.186.190:40839)
0809 15:04:24.412104 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:24.412110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:24.412774 (+ 664us) server_negotiation.cc:407] Connection header received
0809 15:04:24.414543 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:24.414548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:24.414625 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:24.414734 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:24.416083 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.417035 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.418534 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.419069 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.419188 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.422331 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:24.422337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:24.422341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:24.422377 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:24.424868 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:24.425723 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:24.425727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:24.425729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:24.425862 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:24.426315 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:24.426318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:24.426320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:24.426570 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:24.426703 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:24.427417 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 15:04:24.427590 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:04:24.429018 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.411769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:59377)
0809 15:04:24.412602 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:24.412646 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:24.414296 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:24.414907 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:24.414914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:24.415383 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:24.415916 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.415933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.417155 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.417158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.418396 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.418408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.419247 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.419252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.419447 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.420036 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:24.420056 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:24.422118 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:24.425013 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:24.425022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:24.425035 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:24.425591 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:24.425962 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:24.425966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:24.425970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:24.426218 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:24.426701 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:24.426708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:24.427062 (+ 354us) client_negotiation.cc:769] Sending connection context
0809 15:04:24.427327 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 15:04:24.427787 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260809 15:04:24.430292 18939 heartbeater.cc:347] Connected to a master server at 127.27.186.190:40839
I20260809 15:04:24.430572 18939 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:24.431248 18939 heartbeater.cc:511] Master 127.27.186.190:40839 requested a full tablet report, sending...
I20260809 15:04:24.432619 18436 ts_manager.cc:194] Registered new tserver with Master: f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130:33789)
I20260809 15:04:24.433373 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:59377
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:04:24.464927 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:40839
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:24.773242 18945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:24.773473 18945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:24.773584 18945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:24.780478 18945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:24.780568 18945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:24.780715 18945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:04:24.789831 18945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:24.790861 18945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:24.792002 18945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:24.803948 18957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:24.804019 18956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:24.803995 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:24.804687 18945 server_base.cc:1034] running on GCE node
I20260809 15:04:24.805203 18945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:24.806038 18945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:24.807237 18945 hybrid_clock.cc:648] HybridClock initialized: now 1786287864807193 us; error 41 us; skew 500 ppm
I20260809 15:04:24.807505 18945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:24 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:24.810425 18945 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:04:24.811723 18945 webserver.cc:460] Webserver started at http://127.27.186.131:46239/ using document root <none> and password file <none>
I20260809 15:04:24.812274 18945 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:24.812381 18945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:24.812609 18945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:24.815274 18945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "bf94e38aed8d44468706a1020dc53bc5"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "297b9578f4130c34cd5e90c59b5ae026"
server_key_iv: "d0250b28ba3ea1cc834cf58e494d39e7"
server_key_version: "encryptionkey@0"
I20260809 15:04:24.815958 18945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "bf94e38aed8d44468706a1020dc53bc5"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "297b9578f4130c34cd5e90c59b5ae026"
server_key_iv: "d0250b28ba3ea1cc834cf58e494d39e7"
server_key_version: "encryptionkey@0"
I20260809 15:04:24.820539 18945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 15:04:24.823640 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:24.824844 18945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:04:24.824977 18945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "bf94e38aed8d44468706a1020dc53bc5"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "297b9578f4130c34cd5e90c59b5ae026"
server_key_iv: "d0250b28ba3ea1cc834cf58e494d39e7"
server_key_version: "encryptionkey@0"
I20260809 15:04:24.825133 18945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:24.836589 18945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:24.840186 18945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:24.840400 18945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:24.841161 18945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:24.842339 18945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:24.842410 18945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:24.842492 18945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:24.842533 18945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:24.872480 18945 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:39085
I20260809 15:04:24.872532 19141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:39085 every 8 connection(s)
I20260809 15:04:24.873543 18945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:04:24.874255 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 18945
I20260809 15:04:24.874434 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:04:24.875026 28394 external_mini_cluster.cc:1428] Setting key 0351bf52de39261ee774baefb170ca0c
Aug 09 15:04:24 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:24.891800 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.875979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37857 (local address 127.27.186.190:40839)
0809 15:04:24.876162 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:24.876168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:24.877525 (+ 1357us) server_negotiation.cc:407] Connection header received
0809 15:04:24.879247 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:24.879251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:24.879313 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:24.879383 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:24.880678 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.881413 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.882983 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.883433 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.883522 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.886529 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:24.886534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:24.886537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:24.886568 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:24.889389 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:24.890005 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:24.890009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:24.890011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:24.890164 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:24.890512 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:24.890515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:24.890517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:24.890770 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:24.890880 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:24.891401 (+ 521us) server_negotiation.cc:299] Negotiation successful
0809 15:04:24.891565 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:04:24.892590 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.876568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:37857)
0809 15:04:24.877365 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:24.877415 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:24.879012 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:24.879535 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:24.879541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:24.879987 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:24.880533 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.880552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.881537 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.881542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.882854 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.882866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.883600 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.883605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.883788 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.884283 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:24.884302 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:24.886301 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:24.889507 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:24.889513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:24.889522 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:24.889905 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:24.890247 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:24.890250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:24.890252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:24.890414 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:24.890856 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:24.890860 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:24.891106 (+ 246us) client_negotiation.cc:769] Sending connection context
0809 15:04:24.891304 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 15:04:24.891616 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":319,"threads_started":1}
I20260809 15:04:24.893748 19142 heartbeater.cc:347] Connected to a master server at 127.27.186.190:40839
I20260809 15:04:24.894011 19142 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:24.894690 19142 heartbeater.cc:511] Master 127.27.186.190:40839 requested a full tablet report, sending...
I20260809 15:04:24.895932 18436 ts_manager.cc:194] Registered new tserver with Master: bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085)
I20260809 15:04:24.896661 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:37857
I20260809 15:04:24.901615 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:04:24.916674 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.905759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.27.186.190:40839)
0809 15:04:24.905930 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:24.905936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:24.906319 (+ 383us) server_negotiation.cc:407] Connection header received
0809 15:04:24.906443 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:24.906446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:24.906497 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:24.906568 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:24.907799 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.908461 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.909661 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.910046 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.910138 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.912308 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:24.912312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:24.912315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:24.912336 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:24.914552 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:24.915173 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:24.915176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:24.915179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:24.915302 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:24.915586 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:24.915589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:24.915590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:24.915848 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:24.916019 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:24.916264 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 15:04:24.916414 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:04:24.920061 18436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:04:24.926492 18436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:04:24.959717 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.937953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33789 (local address 127.0.0.1:42994)
0809 15:04:24.950336 (+ 12383us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:24.950375 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:24.950522 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:24.952376 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:24.952383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:24.952408 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:24.952808 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.952821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.954596 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.954601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.956758 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.956773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.958630 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.958636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.958854 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.958910 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 15:04:24.959027 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 15:04:24.959250 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12174,"mutex_wait_us":25,"thread_start_us":15924,"threads_started":1}
I20260809 15:04:24.959965 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.937992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39085 (local address 127.0.0.1:43040)
0809 15:04:24.952282 (+ 14290us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:24.952298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:24.952397 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:24.953462 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:24.953467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:24.953492 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:24.953892 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.953904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.955820 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.955825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.957349 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.957366 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.959182 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.959190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.959348 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.959371 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:04:24.959463 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 15:04:24.959587 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14173,"thread_start_us":11955,"threads_started":1}
I20260809 15:04:24.960034 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.938311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.27.186.131:39085)
0809 15:04:24.952915 (+ 14604us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:24.952922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:24.952952 (+ 30us) server_negotiation.cc:407] Connection header received
0809 15:04:24.953035 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:24.953041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:24.953258 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:24.954154 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:24.954388 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.955707 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.957541 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.958942 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.959112 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.959264 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:24.959538 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 15:04:24.959692 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14448,"thread_start_us":14293,"threads_started":1}
I20260809 15:04:24.960784 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.937988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.27.186.130:33789)
0809 15:04:24.951551 (+ 13563us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:24.951559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:24.951588 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:04:24.951705 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:24.951711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:24.951924 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:24.952124 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:24.953003 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.954485 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.956930 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.958440 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.958710 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.958862 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:24.960313 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0809 15:04:24.960466 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13427,"thread_start_us":13281,"threads_started":1}
I20260809 15:04:24.960906 18722 heartbeater.cc:503] Master 127.27.186.190:40839 was elected leader, sending a full tablet report...
I20260809 15:04:24.963194 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.939463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.27.186.129:45865)
0809 15:04:24.951893 (+ 12430us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:24.951900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:24.954348 (+ 2448us) server_negotiation.cc:407] Connection header received
0809 15:04:24.954614 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:24.954621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:24.954859 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:24.955036 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:24.955755 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.957248 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.959493 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.961145 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.961359 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.961512 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:24.962673 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0809 15:04:24.962846 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12280,"thread_start_us":12158,"threads_started":1}
I20260809 15:04:24.963766 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:24.938899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:45950)
0809 15:04:24.954240 (+ 15341us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:24.954255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:24.954451 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:24.955051 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:24.955056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:24.955086 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:24.955559 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.955571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.957401 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.957406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.959309 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:24.959322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:24.961654 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:24.961661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:24.962469 (+ 808us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:24.962502 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:04:24.963202 (+ 700us) client_negotiation.cc:240] Negotiation successful
0809 15:04:24.963411 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15205,"mutex_wait_us":41,"thread_start_us":13130,"threads_started":1}
I20260809 15:04:24.964160 19043 tablet_service.cc:1467] Processing CreateTablet for tablet 0713977016aa405796d5cfa4cba35c4b (DEFAULT_TABLE table=test-table [id=59b918363d5a40b29d9d53562fd6b45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:24.964601 18827 tablet_service.cc:1467] Processing CreateTablet for tablet 0713977016aa405796d5cfa4cba35c4b (DEFAULT_TABLE table=test-table [id=59b918363d5a40b29d9d53562fd6b45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:24.965505 19043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0713977016aa405796d5cfa4cba35c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:24.965727 18827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0713977016aa405796d5cfa4cba35c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:24.967268 18616 tablet_service.cc:1467] Processing CreateTablet for tablet 0713977016aa405796d5cfa4cba35c4b (DEFAULT_TABLE table=test-table [id=59b918363d5a40b29d9d53562fd6b45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:24.968561 18616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0713977016aa405796d5cfa4cba35c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:24.976539 19170 tablet_bootstrap.cc:492] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Bootstrap starting.
I20260809 15:04:24.979417 19171 tablet_bootstrap.cc:492] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Bootstrap starting.
I20260809 15:04:24.980603 19170 tablet_bootstrap.cc:654] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:24.981839 19172 tablet_bootstrap.cc:492] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Bootstrap starting.
I20260809 15:04:24.982115 19170 log.cc:824] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:24.982931 19171 tablet_bootstrap.cc:654] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:24.984036 19171 log.cc:824] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:24.985152 19170 tablet_bootstrap.cc:492] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: No bootstrap required, opened a new log
I20260809 15:04:24.985195 19172 tablet_bootstrap.cc:654] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:24.985409 19170 ts_tablet_manager.cc:1397] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 15:04:24.986245 19172 log.cc:824] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:24.987002 19171 tablet_bootstrap.cc:492] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: No bootstrap required, opened a new log
I20260809 15:04:24.987257 19171 ts_tablet_manager.cc:1397] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260809 15:04:24.988912 19172 tablet_bootstrap.cc:492] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: No bootstrap required, opened a new log
I20260809 15:04:24.989130 19172 ts_tablet_manager.cc:1397] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260809 15:04:24.995492 19172 raft_consensus.cc:348] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:24.995798 19172 raft_consensus.cc:374] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:24.995893 19172 raft_consensus.cc:729] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921308cf39864144b8bed4446d8cf36a, State: Initialized, Role: FOLLOWER
I20260809 15:04:24.995775 19170 raft_consensus.cc:348] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:24.996181 19170 raft_consensus.cc:374] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:24.996318 19170 raft_consensus.cc:729] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35b6bc5b76144929d9f0fc6cc585393, State: Initialized, Role: FOLLOWER
I20260809 15:04:24.996511 19172 consensus_queue.cc:260] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:24.996879 19170 consensus_queue.cc:260] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:24.997186 19171 raft_consensus.cc:348] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:24.997655 19171 raft_consensus.cc:374] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:24.997802 19171 raft_consensus.cc:729] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf94e38aed8d44468706a1020dc53bc5, State: Initialized, Role: FOLLOWER
I20260809 15:04:24.998540 19171 consensus_queue.cc:260] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:24.998217 19172 ts_tablet_manager.cc:1428] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260809 15:04:25.002694 19142 heartbeater.cc:503] Master 127.27.186.190:40839 was elected leader, sending a full tablet report...
I20260809 15:04:25.003301 19171 ts_tablet_manager.cc:1428] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260809 15:04:25.004207 18939 heartbeater.cc:503] Master 127.27.186.190:40839 was elected leader, sending a full tablet report...
I20260809 15:04:25.004627 19170 ts_tablet_manager.cc:1428] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
W20260809 15:04:25.125166 19143 tablet.cc:2369] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:04:25.160727 18940 tablet.cc:2369] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:04:25.184729 18725 tablet.cc:2369] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:25.225931 19180 raft_consensus.cc:482] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:25.226222 19180 raft_consensus.cc:504] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:25.227886 19180 leader_election.cc:288] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085), 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865)
I20260809 15:04:25.236418 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.228912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:39753 (local address 127.27.186.129:45865)
0809 15:04:25.229096 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:25.229100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:25.229248 (+ 148us) server_negotiation.cc:407] Connection header received
0809 15:04:25.229386 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:25.229391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:25.229456 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:25.229558 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:25.230383 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.231468 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.234198 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.235412 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.235556 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.235629 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:25.236051 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 15:04:25.236176 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:04:25.236418 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.228357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:50843 (local address 127.27.186.131:39085)
0809 15:04:25.228744 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:25.228752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:25.229085 (+ 333us) server_negotiation.cc:407] Connection header received
0809 15:04:25.229367 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:25.229372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:25.229452 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:25.229548 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:25.230381 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.231473 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.234198 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.235398 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.235561 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.235629 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:25.236051 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 15:04:25.236176 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260809 15:04:25.236605 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.228091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39085 (local address 127.27.186.130:50843)
0809 15:04:25.228969 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:25.228989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:25.229211 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:25.229636 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:25.229640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:25.229673 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:25.230224 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:25.230243 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.231604 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.231610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:25.234061 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:25.234070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.235665 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.235670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:25.235877 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.235924 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 15:04:25.236004 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 15:04:25.236180 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":173,"thread_start_us":429,"threads_started":1}
I20260809 15:04:25.236678 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.228357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.27.186.130:39753)
0809 15:04:25.229165 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:25.229180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:25.229283 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:25.229633 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:25.229647 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:25.229731 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:25.230259 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:25.230269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.231608 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.231612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:25.234052 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:25.234068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.235659 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.235665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:25.235861 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.235913 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 15:04:25.236004 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 15:04:25.236165 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":72,"spinlock_wait_cycles":22400,"thread_start_us":484,"threads_started":1}
I20260809 15:04:25.237587 19075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0713977016aa405796d5cfa4cba35c4b" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf94e38aed8d44468706a1020dc53bc5" is_pre_election: true
I20260809 15:04:25.237586 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0713977016aa405796d5cfa4cba35c4b" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921308cf39864144b8bed4446d8cf36a" is_pre_election: true
I20260809 15:04:25.238087 18650 raft_consensus.cc:2454] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 0.
I20260809 15:04:25.238083 19075 raft_consensus.cc:2454] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 0.
I20260809 15:04:25.238752 18759 leader_election.cc:302] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 921308cf39864144b8bed4446d8cf36a, bf94e38aed8d44468706a1020dc53bc5, f35b6bc5b76144929d9f0fc6cc585393; no voters:
I20260809 15:04:25.239133 19180 raft_consensus.cc:2781] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:04:25.239310 19180 raft_consensus.cc:482] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:04:25.239413 19180 raft_consensus.cc:3037] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:25.241521 19180 raft_consensus.cc:504] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:25.242063 19180 leader_election.cc:288] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 election: Requested vote from peers bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085), 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865)
I20260809 15:04:25.242486 19075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0713977016aa405796d5cfa4cba35c4b" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf94e38aed8d44468706a1020dc53bc5"
I20260809 15:04:25.242527 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0713977016aa405796d5cfa4cba35c4b" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921308cf39864144b8bed4446d8cf36a"
I20260809 15:04:25.242690 18650 raft_consensus.cc:3037] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:25.242690 19075 raft_consensus.cc:3037] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:25.245759 19075 raft_consensus.cc:2454] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 1.
I20260809 15:04:25.245759 18650 raft_consensus.cc:2454] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 1.
I20260809 15:04:25.246160 18759 leader_election.cc:302] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf94e38aed8d44468706a1020dc53bc5, f35b6bc5b76144929d9f0fc6cc585393; no voters:
I20260809 15:04:25.246408 19180 raft_consensus.cc:2781] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:25.246830 19180 raft_consensus.cc:686] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 1 LEADER]: Becoming Leader. State: Replica: f35b6bc5b76144929d9f0fc6cc585393, State: Running, Role: LEADER
I20260809 15:04:25.247294 19180 consensus_queue.cc:237] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } }
I20260809 15:04:25.251794 18434 catalog_manager.cc:5495] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 reported cstate change: term changed from 0 to 1, leader changed from <none> to f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130). New cstate: current_term: 1 leader_uuid: "f35b6bc5b76144929d9f0fc6cc585393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } health_report { overall_health: UNKNOWN } } }
I20260809 15:04:25.321348 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.316282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.27.186.130:33789)
0809 15:04:25.316452 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:25.316456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:25.316587 (+ 131us) server_negotiation.cc:407] Connection header received
0809 15:04:25.316781 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:25.316785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:25.316869 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:25.316988 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:04:25.317304 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.318043 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.319543 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.319940 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.320023 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.320464 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:04:25.320929 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:04:25.321029 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:25.321091 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 15:04:25.321151 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:04:25.329432 19075 raft_consensus.cc:1261] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Refusing update from remote peer f35b6bc5b76144929d9f0fc6cc585393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:25.329432 18650 raft_consensus.cc:1261] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Refusing update from remote peer f35b6bc5b76144929d9f0fc6cc585393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:25.330478 19186 consensus_queue.cc:1035] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:25.330847 19180 consensus_queue.cc:1035] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:25.346274 19190 mvcc.cc:205] Tried to move back new op lower bound from 7316635096378724352 to 7316635096056741888. Current Snapshot: MvccSnapshot[applied={T|T < 7316635096378724352}]
I20260809 15:04:25.364759 18434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:25.365036 18434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:25.368278 18434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Sending DeleteTablet for 3 replicas of tablet 0713977016aa405796d5cfa4cba35c4b
I20260809 15:04:25.369204 19043 tablet_service.cc:1514] Processing DeleteTablet for tablet 0713977016aa405796d5cfa4cba35c4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:25 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:43040
I20260809 15:04:25.369333 18827 tablet_service.cc:1514] Processing DeleteTablet for tablet 0713977016aa405796d5cfa4cba35c4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:25 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:42994
I20260809 15:04:25.370366 18616 tablet_service.cc:1514] Processing DeleteTablet for tablet 0713977016aa405796d5cfa4cba35c4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:25 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45950
I20260809 15:04:25.370477 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 18528
I20260809 15:04:25.380757 19210 tablet_replica.cc:330] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: stopping tablet replica
I20260809 15:04:25.380956 19207 tablet_replica.cc:330] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: stopping tablet replica
I20260809 15:04:25.381220 19210 raft_consensus.cc:2229] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:04:25.381369 19207 raft_consensus.cc:2229] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:25.381922 19207 raft_consensus.cc:2258] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:25.383363 19210 raft_consensus.cc:2258] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:25.384290 19207 ts_tablet_manager.cc:1905] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:25.385412 19210 ts_tablet_manager.cc:1905] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:25.392117 19207 ts_tablet_manager.cc:1918] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:25.392249 19207 log.cc:1196] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/0713977016aa405796d5cfa4cba35c4b
I20260809 15:04:25.392678 19207 ts_tablet_manager.cc:1939] T 0713977016aa405796d5cfa4cba35c4b P bf94e38aed8d44468706a1020dc53bc5: Deleting consensus metadata
I20260809 15:04:25.394096 18414 catalog_manager.cc:4841] TS bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085): tablet 0713977016aa405796d5cfa4cba35c4b (table test-table [id=59b918363d5a40b29d9d53562fd6b45b]) successfully deleted
I20260809 15:04:25.394304 19212 tablet_replica.cc:330] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: stopping tablet replica
I20260809 15:04:25.394629 19212 raft_consensus.cc:2229] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Raft consensus shutting down.
Aug 09 15:04:25 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:04:25.395023 19212 raft_consensus.cc:2258] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:25.396956 19212 ts_tablet_manager.cc:1905] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:25.399281 19210 ts_tablet_manager.cc:1918] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:25.399539 19210 log.cc:1196] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/0713977016aa405796d5cfa4cba35c4b
I20260809 15:04:25.400225 19210 ts_tablet_manager.cc:1939] T 0713977016aa405796d5cfa4cba35c4b P f35b6bc5b76144929d9f0fc6cc585393: Deleting consensus metadata
I20260809 15:04:25.402863 18413 catalog_manager.cc:4841] TS f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130:33789): tablet 0713977016aa405796d5cfa4cba35c4b (table test-table [id=59b918363d5a40b29d9d53562fd6b45b]) successfully deleted
I20260809 15:04:25.403821 19212 ts_tablet_manager.cc:1918] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:25.404047 19212 log.cc:1196] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/0713977016aa405796d5cfa4cba35c4b
I20260809 15:04:25.404606 19212 ts_tablet_manager.cc:1939] T 0713977016aa405796d5cfa4cba35c4b P 921308cf39864144b8bed4446d8cf36a: Deleting consensus metadata
I20260809 15:04:25.404917 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.371671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.27.186.129:45865)
0809 15:04:25.371837 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:25.371843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:25.382873 (+ 11030us) server_negotiation.cc:407] Connection header received
0809 15:04:25.383180 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:25.383186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:25.383296 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:25.383378 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:25.385779 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.387217 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.389298 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.390045 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.390169 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.396690 (+ 6521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:25.396702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:25.396718 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:25.396761 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:25.400491 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:25.401408 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:25.401417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:25.401421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:25.401589 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:25.402100 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:25.402105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:25.402108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:25.402626 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:25.402941 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:25.404142 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0809 15:04:25.404384 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:04:25.405652 18712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:25.406283 18412 catalog_manager.cc:4841] TS 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865): tablet 0713977016aa405796d5cfa4cba35c4b (table test-table [id=59b918363d5a40b29d9d53562fd6b45b]) successfully deleted
I20260809 15:04:25.502753 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 18528
I20260809 15:04:25.534574 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 18732
Aug 09 15:04:25 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:04:25.550289 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.535396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.27.186.130:33789)
0809 15:04:25.535616 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:25.535622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:25.535643 (+ 21us) server_negotiation.cc:407] Connection header received
0809 15:04:25.535792 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:25.535797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:25.535881 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:25.535967 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:25.537250 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.538109 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.539414 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.540101 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.540220 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.544477 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:25.544489 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:25.544503 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:25.544537 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:25.547512 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:25.548208 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:25.548215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:25.548217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:25.548365 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:25.548739 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:25.548743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:25.548745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:25.549094 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:25.549424 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:25.549715 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 15:04:25.549930 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 15:04:25.550981 18929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:25.659904 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 18732
I20260809 15:04:25.684798 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 18945
Aug 09 15:04:25 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:04:25.699929 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:25.685470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.27.186.131:39085)
0809 15:04:25.685631 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:25.685636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:25.685701 (+ 65us) server_negotiation.cc:407] Connection header received
0809 15:04:25.685864 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:25.685868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:25.685943 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:25.686040 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:25.687269 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.687967 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.689457 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:25.689883 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:25.689999 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:25.694190 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:25.694206 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:25.694221 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:25.694257 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:25.697172 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:25.697804 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:25.697811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:25.697814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:25.697987 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:25.698484 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:25.698488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:25.698490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:25.698854 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:25.699133 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:25.699416 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 15:04:25.699622 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 15:04:25.700552 19134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:25.785429 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 18945
I20260809 15:04:25.807715 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 18386
I20260809 15:04:25.808832 18485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:25.914981 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 18386
I20260809 15:04:25.939380 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40839
--webserver_interface=127.27.186.190
--webserver_port=45309
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:26.251986 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:26.252280 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:26.252382 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:26.259449 19219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:26.259545 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:26.259614 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:26.259684 19219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:26.259728 19219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:26.269016 19219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=45309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:26.270215 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:26.271415 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:26.293687 19229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:26.293610 19233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:26.293643 19230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:26.294504 19219 server_base.cc:1034] running on GCE node
I20260809 15:04:26.295336 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:26.296729 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:26.297935 19219 hybrid_clock.cc:648] HybridClock initialized: now 1786287866297881 us; error 40 us; skew 500 ppm
I20260809 15:04:26.298341 19219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:26 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287866, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:26.304716 19219 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:26.306239 19219 webserver.cc:460] Webserver started at http://127.27.186.190:45309/ using document root <none> and password file <none>
I20260809 15:04:26.306929 19219 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:26.307140 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:26.312927 19219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 15:04:26.315738 19240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:26.316952 19219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 15:04:26.317132 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "51cf2597223245ff8a975aa72b946ccd"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "1bd4c417305110d654d0f1a28328d41b"
server_key_iv: "b4af4ebe527dcb77b30580891c6e2c73"
server_key_version: "encryptionkey@0"
I20260809 15:04:26.318157 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:26.338160 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:26.341771 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:26.342008 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:26.359074 19219 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:40839
I20260809 15:04:26.359091 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:40839 every 8 connection(s)
I20260809 15:04:26.360423 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:26.360837 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 19219
I20260809 15:04:26.361814 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:45865
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:40839
--webserver_port=39857
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:26.368916 19319 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:04:26.373025 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap starting.
I20260809 15:04:26.391187 19319 log.cc:824] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:26.402624 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:04:26.403085 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap complete.
I20260809 15:04:26.411576 19319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:26.412344 19319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cf2597223245ff8a975aa72b946ccd, State: Initialized, Role: FOLLOWER
I20260809 15:04:26.412974 19319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:26.413197 19319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:26.413311 19319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:26.413466 19319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:04:26.416998 19319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:26.417483 19319 leader_election.cc:302] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cf2597223245ff8a975aa72b946ccd; no voters:
I20260809 15:04:26.418038 19319 leader_election.cc:288] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 15:04:26.418208 19326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:04:26.419231 19326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 LEADER]: Becoming Leader. State: Replica: 51cf2597223245ff8a975aa72b946ccd, State: Running, Role: LEADER
I20260809 15:04:26.419903 19319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:26.419817 19326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:26.425803 19327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "51cf2597223245ff8a975aa72b946ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } } }
I20260809 15:04:26.426170 19327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:26.426713 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cf2597223245ff8a975aa72b946ccd. Latest consensus state: current_term: 2 leader_uuid: "51cf2597223245ff8a975aa72b946ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } } }
I20260809 15:04:26.426981 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:26.433218 19353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:26.440227 19353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:26.441824 19350 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 15:04:26.442049 19353 catalog_manager.cc:1259] Loaded cluster ID: 4f4712e8f2ce475393f3682c5e4bcd9a
I20260809 15:04:26.442195 19353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:26.445657 19353 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 15:04:26.462381 19353 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 15:04:26.470635 19353 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:26.472165 19353 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Loaded TSK: 0
I20260809 15:04:26.473521 19353 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 15:04:26.695286 19322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:26.695531 19322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:26.695662 19322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:26.702592 19322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:26.702698 19322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:26.702839 19322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:04:26.712030 19322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=39857
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:26.713084 19322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:26.714344 19322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:26.728235 19367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:26.728333 19366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:26.728863 19371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:26.731308 19322 server_base.cc:1034] running on GCE node
I20260809 15:04:26.731838 19322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:26.732707 19322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:26.733917 19322 hybrid_clock.cc:648] HybridClock initialized: now 1786287866733864 us; error 49 us; skew 500 ppm
I20260809 15:04:26.734234 19322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:26 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287866, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:26.737371 19322 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:04:26.738858 19322 webserver.cc:460] Webserver started at http://127.27.186.129:39857/ using document root <none> and password file <none>
I20260809 15:04:26.739490 19322 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:26.739625 19322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:26.744656 19322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:04:26.747263 19379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:26.748523 19322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:26.748668 19322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "921308cf39864144b8bed4446d8cf36a"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "0399b6f9db88c79851bf355b9111877d"
server_key_iv: "b0dbc7e1a8a32db9224a29e06c2e72b3"
server_key_version: "encryptionkey@0"
I20260809 15:04:26.749563 19322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:26.763571 19322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:26.767272 19322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:26.767518 19322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:26.768461 19322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:26.769804 19322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:26.769896 19322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:26.769995 19322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:26.770095 19322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 15:04:26.800825 19322 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:45865
I20260809 15:04:26.800905 19549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:45865 every 8 connection(s)
I20260809 15:04:26.802171 19322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:04:26.808368 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 19322
I20260809 15:04:26.809118 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:33789
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:40839
--webserver_port=46385
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:04:26 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287866, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:26.824091 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:26.804935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:59297)
0809 15:04:26.805689 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:26.805746 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:26.807401 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:26.808642 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:26.808652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:26.809324 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:26.809991 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:26.810040 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:26.811633 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:26.811638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:26.813345 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:26.813361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:26.814335 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:26.814343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:26.814593 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:26.815236 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:26.815258 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:26.817273 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:26.820275 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:26.820283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:26.820292 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:26.820672 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:26.821123 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:26.821146 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:26.821150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:26.821372 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:26.822109 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:26.822117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:26.822470 (+ 353us) client_negotiation.cc:769] Sending connection context
0809 15:04:26.822732 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 15:04:26.823122 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":323,"threads_started":1}
I20260809 15:04:26.824390 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:26.804741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:59297 (local address 127.27.186.190:40839)
0809 15:04:26.805412 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:26.805423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:26.805847 (+ 424us) server_negotiation.cc:407] Connection header received
0809 15:04:26.807839 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:26.807852 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:26.808162 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:26.808441 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:26.810227 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:26.811476 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:26.813465 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:26.814120 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:26.814246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:26.817511 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:26.817519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:26.817531 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:26.817564 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:26.820131 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:26.820802 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:26.820808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:26.820813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:26.821017 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:26.821469 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:26.821474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:26.821478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:26.821968 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:26.822171 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:26.822970 (+ 799us) server_negotiation.cc:299] Negotiation successful
0809 15:04:26.823344 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":322,"threads_started":1}
I20260809 15:04:26.826920 19552 heartbeater.cc:347] Connected to a master server at 127.27.186.190:40839
I20260809 15:04:26.827216 19552 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:26.827982 19552 heartbeater.cc:511] Master 127.27.186.190:40839 requested a full tablet report, sending...
I20260809 15:04:26.830135 19263 ts_manager.cc:194] Registered new tserver with Master: 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865)
I20260809 15:04:26.831492 19263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:59297
W20260809 15:04:27.132709 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:27.132958 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:27.133111 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:27.140511 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:27.140612 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:27.140770 19557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:04:27.149887 19557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:33789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=46385
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:27.150950 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:27.152086 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:27.164103 19565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:27.164172 19562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:27.164120 19563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:27.164805 19557 server_base.cc:1034] running on GCE node
I20260809 15:04:27.165333 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:27.166199 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:27.167402 19557 hybrid_clock.cc:648] HybridClock initialized: now 1786287867167346 us; error 44 us; skew 500 ppm
I20260809 15:04:27.167680 19557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:27 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287867, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:27.170542 19557 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:04:27.171845 19557 webserver.cc:460] Webserver started at http://127.27.186.130:46385/ using document root <none> and password file <none>
I20260809 15:04:27.172434 19557 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:27.172564 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:27.177472 19557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:04:27.179973 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:27.181130 19557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:04:27.181284 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "f35b6bc5b76144929d9f0fc6cc585393"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "ca0522b22d19a5f433eeea6e48b4774d"
server_key_iv: "817ec646be44412c81fed928cdc0050e"
server_key_version: "encryptionkey@0"
I20260809 15:04:27.182200 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:27.203454 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:27.207439 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:27.207639 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:27.208453 19557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:27.209612 19557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:27.209698 19557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:27.209786 19557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:27.209839 19557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:27.237799 19557 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33789
I20260809 15:04:27.237861 19745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33789 every 8 connection(s)
I20260809 15:04:27.239064 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:04:27.246378 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 19557
I20260809 15:04:27.247152 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:39085
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:40839
--webserver_port=46239
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:04:27 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287867, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:27.258671 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.241296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56223 (local address 127.27.186.190:40839)
0809 15:04:27.241468 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.241472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.242507 (+ 1035us) server_negotiation.cc:407] Connection header received
0809 15:04:27.244272 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.244278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.244361 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.244453 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:27.245888 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.246944 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.248664 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.249269 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.249392 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.252992 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:27.252999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:27.253004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:27.253045 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:27.255927 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:27.256574 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:27.256579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:27.256582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:27.256730 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:27.257081 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:27.257085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:27.257087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:27.257414 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:27.257556 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.258255 (+ 699us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.258424 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 15:04:27.259958 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.241660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:56223)
0809 15:04:27.242354 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.242392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.244008 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.244707 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.244715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.245170 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:27.245716 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.245734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.247127 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.247131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.248491 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.248505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.249520 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.249526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.249759 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.250425 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:27.250459 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:27.252774 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:27.256054 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:27.256061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:27.256072 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:27.256466 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:27.256818 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:27.256821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:27.256823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:27.256989 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:27.257516 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:27.257521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:27.257849 (+ 328us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.258169 (+ 320us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.258660 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":283,"threads_started":1}
I20260809 15:04:27.261288 19746 heartbeater.cc:347] Connected to a master server at 127.27.186.190:40839
I20260809 15:04:27.261575 19746 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:27.262334 19746 heartbeater.cc:511] Master 127.27.186.190:40839 requested a full tablet report, sending...
I20260809 15:04:27.263892 19263 ts_manager.cc:194] Registered new tserver with Master: f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130:33789)
I20260809 15:04:27.264669 19263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:56223
W20260809 15:04:27.567027 19754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:27.567281 19754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:27.567423 19754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:27.574285 19754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:27.574389 19754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:27.574532 19754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:04:27.583477 19754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:39085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=46239
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:27.584542 19754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:27.585793 19754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:27.598081 19764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:27.598089 19762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:27.598011 19766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:27.598806 19754 server_base.cc:1034] running on GCE node
I20260809 15:04:27.599380 19754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:27.600214 19754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:27.601405 19754 hybrid_clock.cc:648] HybridClock initialized: now 1786287867601361 us; error 33 us; skew 500 ppm
I20260809 15:04:27.601691 19754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:27 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287867, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:27.604738 19754 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:04:27.606098 19754 webserver.cc:460] Webserver started at http://127.27.186.131:46239/ using document root <none> and password file <none>
I20260809 15:04:27.606740 19754 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:27.606871 19754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:27.611737 19754 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:04:27.614269 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:27.615510 19754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:27.615660 19754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "bf94e38aed8d44468706a1020dc53bc5"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "297b9578f4130c34cd5e90c59b5ae026"
server_key_iv: "d0250b28ba3ea1cc834cf58e494d39e7"
server_key_version: "encryptionkey@0"
I20260809 15:04:27.616565 19754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:27.668407 19754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:27.672063 19754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:27.672303 19754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:27.673219 19754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:27.674500 19754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:27.674600 19754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:27.674702 19754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:27.674772 19754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:27.704788 19754 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:39085
I20260809 15:04:27.704867 19983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:39085 every 8 connection(s)
I20260809 15:04:27.706112 19754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:04:27.715049 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 19754
Aug 09 15:04:27 dist-test-slave-q9h9 krb5kdc[18361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287867, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:27.726589 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.708404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:45229 (local address 127.27.186.190:40839)
0809 15:04:27.708578 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.708582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.709636 (+ 1054us) server_negotiation.cc:407] Connection header received
0809 15:04:27.711397 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.711402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.711478 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.711566 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:27.712958 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.713684 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.715532 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.716154 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.716298 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.720430 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:27.720436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:27.720439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:27.720473 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:27.723444 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:27.724317 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:27.724321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:27.724324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:27.724438 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:27.724883 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:27.724885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:27.724887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:27.725131 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:27.725321 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.725920 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.726218 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:04:27.727329 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.708829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:45229)
0809 15:04:27.709495 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.709530 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.711143 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.711808 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.711816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.712273 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:27.712792 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.712809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.713831 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.713835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.715389 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.715405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.716595 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.716602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.716855 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.717521 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:27.717550 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:27.720196 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:27.723606 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:27.723615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:27.723630 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:27.724193 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:27.724538 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:27.724543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:27.724547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:27.724788 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:27.725290 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:27.725295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:27.725565 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.725789 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.726278 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":255,"threads_started":1}
I20260809 15:04:27.728632 19985 heartbeater.cc:347] Connected to a master server at 127.27.186.190:40839
I20260809 15:04:27.728899 19985 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:27.729537 19985 heartbeater.cc:511] Master 127.27.186.190:40839 requested a full tablet report, sending...
I20260809 15:04:27.730592 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.715976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.27.186.190:40839)
0809 15:04:27.716440 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.716444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.716886 (+ 442us) server_negotiation.cc:407] Connection header received
0809 15:04:27.717034 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.717038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.717112 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.717213 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:27.719165 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.720101 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.721649 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.722305 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.722428 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.725236 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:27.725242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:27.725246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:27.725277 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:27.727954 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:27.728719 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:27.728724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:27.728728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:27.728867 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:27.729223 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:27.729228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:27.729232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:27.729481 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:27.729704 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.730057 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.730255 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260809 15:04:27.731172 19263 ts_manager.cc:194] Registered new tserver with Master: bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085)
I20260809 15:04:27.732165 19263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:45229
I20260809 15:04:27.733060 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:04:27.743014 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.733949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.27.186.190:40839)
0809 15:04:27.734102 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.734106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.738451 (+ 4345us) server_negotiation.cc:407] Connection header received
0809 15:04:27.738593 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.738597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.738725 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.738830 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:04:27.739151 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.739784 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.741241 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.741633 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.741726 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.742156 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:04:27.742561 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:04:27.742663 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.742739 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.742793 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:04:27.743808 19261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:04:27.750691 19261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:04:27.783735 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.762138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.27.186.129:45865)
0809 15:04:27.776118 (+ 13980us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.776127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.776157 (+ 30us) server_negotiation.cc:407] Connection header received
0809 15:04:27.776263 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.776271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.776521 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.776712 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.777500 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.779043 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.781092 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.782508 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.782665 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.782825 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.783239 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.783413 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13838,"thread_start_us":13735,"threads_started":1}
I20260809 15:04:27.783739 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.761421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:45974)
0809 15:04:27.770408 (+ 8987us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.770452 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.770625 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.776731 (+ 6106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.776739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.776770 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.777301 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.777316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.779198 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.779204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.780943 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.780956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.782740 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.782747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.782994 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.783055 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.783179 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.783364 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8721,"mutex_wait_us":36,"thread_start_us":15652,"threads_started":1}
I20260809 15:04:27.786295 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.761482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39085 (local address 127.0.0.1:43060)
0809 15:04:27.777391 (+ 15909us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.777410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.777522 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.779127 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.779136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.779165 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.779721 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.779733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.781552 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.781557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.783424 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.783438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.785448 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.785456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.785618 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.785642 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.785727 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.785845 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15775,"mutex_wait_us":85,"thread_start_us":16248,"threads_started":1}
I20260809 15:04:27.786398 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.761644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.27.186.131:39085)
0809 15:04:27.778394 (+ 16750us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.778401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.778431 (+ 30us) server_negotiation.cc:407] Connection header received
0809 15:04:27.778518 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.778525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.778719 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.778880 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.779940 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.781430 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.783610 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.785178 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.785356 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.785514 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.785791 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.786062 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16584,"thread_start_us":16424,"threads_started":1}
I20260809 15:04:27.787127 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.762393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.27.186.130:33789)
0809 15:04:27.775593 (+ 13200us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.775602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.778099 (+ 2497us) server_negotiation.cc:407] Connection header received
0809 15:04:27.778423 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.778431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.778643 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.778838 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.779694 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.781178 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.783568 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.785222 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.785397 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.785575 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.786682 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.786850 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13021,"thread_start_us":13204,"threads_started":1}
I20260809 15:04:27.787622 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.761894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33789 (local address 127.0.0.1:43020)
0809 15:04:27.777975 (+ 16081us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.777991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.778193 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.779055 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.779059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.779082 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.779534 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.779543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.781307 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.781313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.783390 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.783404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.785792 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.785799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.785983 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.786039 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.786124 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.786245 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15978,"mutex_wait_us":46,"spinlock_wait_cycles":15744,"thread_start_us":8111,"threads_started":1}
I20260809 15:04:27.787739 19449 tablet_service.cc:1467] Processing CreateTablet for tablet 54d26cdf4541455190397310ea96599c (DEFAULT_TABLE table=test-table [id=a1e9dc8da78e49ecb88f38673cc8c879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:27.789053 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d26cdf4541455190397310ea96599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:27.790517 19651 tablet_service.cc:1467] Processing CreateTablet for tablet 54d26cdf4541455190397310ea96599c (DEFAULT_TABLE table=test-table [id=a1e9dc8da78e49ecb88f38673cc8c879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:27.791662 19651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d26cdf4541455190397310ea96599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:27.797340 19867 tablet_service.cc:1467] Processing CreateTablet for tablet 54d26cdf4541455190397310ea96599c (DEFAULT_TABLE table=test-table [id=a1e9dc8da78e49ecb88f38673cc8c879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:27.798653 19867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d26cdf4541455190397310ea96599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:27.812564 20012 tablet_bootstrap.cc:492] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Bootstrap starting.
I20260809 15:04:27.812558 20014 tablet_bootstrap.cc:492] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Bootstrap starting.
I20260809 15:04:27.814435 20013 tablet_bootstrap.cc:492] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Bootstrap starting.
I20260809 15:04:27.815992 20014 tablet_bootstrap.cc:654] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:27.816820 20012 tablet_bootstrap.cc:654] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:27.817216 20013 tablet_bootstrap.cc:654] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:27.817267 20014 log.cc:824] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:27.818003 20012 log.cc:824] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:27.818363 20013 log.cc:824] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:27.820230 20014 tablet_bootstrap.cc:492] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: No bootstrap required, opened a new log
I20260809 15:04:27.820541 20014 ts_tablet_manager.cc:1397] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260809 15:04:27.821044 20013 tablet_bootstrap.cc:492] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: No bootstrap required, opened a new log
I20260809 15:04:27.821302 20013 ts_tablet_manager.cc:1397] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:04:27.822100 20012 tablet_bootstrap.cc:492] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: No bootstrap required, opened a new log
I20260809 15:04:27.822345 20012 ts_tablet_manager.cc:1397] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260809 15:04:27.828620 20013 raft_consensus.cc:348] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.828930 20013 raft_consensus.cc:374] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:27.829020 20013 raft_consensus.cc:729] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35b6bc5b76144929d9f0fc6cc585393, State: Initialized, Role: FOLLOWER
I20260809 15:04:27.829571 20013 consensus_queue.cc:260] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.830590 19746 heartbeater.cc:503] Master 127.27.186.190:40839 was elected leader, sending a full tablet report...
I20260809 15:04:27.831036 20013 ts_tablet_manager.cc:1428] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260809 15:04:27.830864 20014 raft_consensus.cc:348] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.831265 20014 raft_consensus.cc:374] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:27.831372 20014 raft_consensus.cc:729] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf94e38aed8d44468706a1020dc53bc5, State: Initialized, Role: FOLLOWER
I20260809 15:04:27.831965 20014 consensus_queue.cc:260] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.832954 19985 heartbeater.cc:503] Master 127.27.186.190:40839 was elected leader, sending a full tablet report...
I20260809 15:04:27.833535 20014 ts_tablet_manager.cc:1428] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 15:04:27.833930 20012 raft_consensus.cc:348] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.834353 20012 raft_consensus.cc:374] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:27.834466 20012 raft_consensus.cc:729] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921308cf39864144b8bed4446d8cf36a, State: Initialized, Role: FOLLOWER
I20260809 15:04:27.834741 19552 heartbeater.cc:503] Master 127.27.186.190:40839 was elected leader, sending a full tablet report...
I20260809 15:04:27.835078 20012 consensus_queue.cc:260] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.836746 20012 ts_tablet_manager.cc:1428] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260809 15:04:27.957594 19986 tablet.cc:2369] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:27.981312 20022 raft_consensus.cc:482] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:27.981598 20022 raft_consensus.cc:504] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.983561 20022 leader_election.cc:288] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085), 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865)
I20260809 15:04:27.989810 20024 raft_consensus.cc:482] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:27.990288 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.984017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:46915 (local address 127.27.186.129:45865)
0809 15:04:27.984188 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.984194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.984903 (+ 709us) server_negotiation.cc:407] Connection header received
0809 15:04:27.985056 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.985060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.985120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.985206 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.985872 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.986628 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.988529 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.989263 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.989424 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.989498 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.989885 (+ 387us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.990006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 15:04:27.990111 20024 raft_consensus.cc:504] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.990528 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.984025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.27.186.130:46915)
0809 15:04:27.984801 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.984817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.984940 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.985240 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.985244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.985275 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.985671 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.985679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.986781 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.986786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.988382 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.988394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.989501 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.989508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.989698 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.989750 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.989841 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.990007 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":38,"thread_start_us":424,"threads_started":1}
I20260809 15:04:27.990672 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.983753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:39085 (local address 127.27.186.130:49919)
0809 15:04:27.984645 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.984659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.984809 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.985178 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.985182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.985209 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.985622 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.985635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.986735 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.986741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.988387 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.988400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.989536 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.989544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.989712 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.989776 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.990148 (+ 372us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.990277 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":102,"thread_start_us":556,"threads_started":1}
W20260809 15:04:27.990849 19750 tablet.cc:2369] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:27.991801 20024 leader_election.cc:288] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865), f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130:33789)
I20260809 15:04:27.992358 19478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d26cdf4541455190397310ea96599c" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921308cf39864144b8bed4446d8cf36a" is_pre_election: true
I20260809 15:04:27.992938 19478 raft_consensus.cc:2454] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 0.
I20260809 15:04:27.993085 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.983955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:49919 (local address 127.27.186.131:39085)
0809 15:04:27.984152 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.984158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.984749 (+ 591us) server_negotiation.cc:407] Connection header received
0809 15:04:27.984987 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.984993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.985071 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.985143 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.985782 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.986608 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.988528 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.989288 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.989427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.989498 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.989933 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.990089 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:04:27.994045 19580 leader_election.cc:302] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 921308cf39864144b8bed4446d8cf36a, f35b6bc5b76144929d9f0fc6cc585393; no voters:
I20260809 15:04:27.993922 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d26cdf4541455190397310ea96599c" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf94e38aed8d44468706a1020dc53bc5" is_pre_election: true
I20260809 15:04:27.994468 20022 raft_consensus.cc:2781] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:04:27.994436 19896 raft_consensus.cc:2454] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 0.
I20260809 15:04:27.994603 20022 raft_consensus.cc:482] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:04:27.994690 20022 raft_consensus.cc:3037] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:27.998044 20022 raft_consensus.cc:504] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:27.998309 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.992416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.27.186.131:37403)
0809 15:04:27.992569 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.992587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.992703 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.993390 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.993395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.993422 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.993608 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.993619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.995211 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.995216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.996557 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.996569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.997613 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.997618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:27.997797 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.997820 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:04:27.997897 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 15:04:27.998052 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 15:04:27.998315 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.992248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:37403 (local address 127.27.186.129:45865)
0809 15:04:27.992383 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.992389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.993112 (+ 723us) server_negotiation.cc:407] Connection header received
0809 15:04:27.993175 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.993180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.993246 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.993761 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.994039 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.995040 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.996722 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:27.997467 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:27.997607 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:27.997681 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:27.997936 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 15:04:27.998076 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 15:04:27.998711 20022 leader_election.cc:288] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 election: Requested vote from peers bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085), 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865)
I20260809 15:04:27.999897 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d26cdf4541455190397310ea96599c" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921308cf39864144b8bed4446d8cf36a"
I20260809 15:04:28.000200 19477 raft_consensus.cc:3037] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:28.003481 19477 raft_consensus.cc:2454] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 1.
I20260809 15:04:28.003938 19580 leader_election.cc:302] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 921308cf39864144b8bed4446d8cf36a, f35b6bc5b76144929d9f0fc6cc585393; no voters:
I20260809 15:04:28.003925 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d26cdf4541455190397310ea96599c" candidate_uuid: "f35b6bc5b76144929d9f0fc6cc585393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf94e38aed8d44468706a1020dc53bc5"
I20260809 15:04:28.004129 19896 raft_consensus.cc:3037] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:28.004226 20022 raft_consensus.cc:2781] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:28.004618 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.992673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:33789 (local address 127.27.186.131:52397)
0809 15:04:27.998500 (+ 5827us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:27.998514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:27.998618 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:27.999035 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:27.999039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:27.999063 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:27.999232 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:27.999242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.000448 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.000453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:28.002240 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:28.002258 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.003806 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.003812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:28.004028 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:28.004060 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:04:28.004165 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 15:04:28.004284 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5737,"thread_start_us":9524,"threads_started":1}
I20260809 15:04:28.004612 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:27.992168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:52397 (local address 127.27.186.130:33789)
0809 15:04:27.992350 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:27.992355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:27.998586 (+ 6231us) server_negotiation.cc:407] Connection header received
0809 15:04:27.998723 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:27.998727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:27.998818 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:27.998893 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:27.999374 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.000270 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.002465 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.003630 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.003777 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:28.003864 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:28.004180 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 15:04:28.004393 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:04:28.004943 20022 raft_consensus.cc:686] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 1 LEADER]: Becoming Leader. State: Replica: f35b6bc5b76144929d9f0fc6cc585393, State: Running, Role: LEADER
I20260809 15:04:28.005286 19478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d26cdf4541455190397310ea96599c" candidate_uuid: "bf94e38aed8d44468706a1020dc53bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921308cf39864144b8bed4446d8cf36a" is_pre_election: true
I20260809 15:04:28.005534 19478 raft_consensus.cc:2379] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf94e38aed8d44468706a1020dc53bc5 in current term 1: Already voted for candidate f35b6bc5b76144929d9f0fc6cc585393 in this term.
I20260809 15:04:28.005475 20022 consensus_queue.cc:237] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } }
I20260809 15:04:28.007558 19896 raft_consensus.cc:2454] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35b6bc5b76144929d9f0fc6cc585393 in term 1.
I20260809 15:04:28.008566 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d26cdf4541455190397310ea96599c" candidate_uuid: "bf94e38aed8d44468706a1020dc53bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35b6bc5b76144929d9f0fc6cc585393" is_pre_election: true
I20260809 15:04:28.009462 19787 leader_election.cc:302] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf94e38aed8d44468706a1020dc53bc5; no voters: 921308cf39864144b8bed4446d8cf36a, f35b6bc5b76144929d9f0fc6cc585393
I20260809 15:04:28.009812 20024 raft_consensus.cc:2726] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:04:28.010996 19263 catalog_manager.cc:5495] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 reported cstate change: term changed from 0 to 1, leader changed from <none> to f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130). New cstate: current_term: 1 leader_uuid: "f35b6bc5b76144929d9f0fc6cc585393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f35b6bc5b76144929d9f0fc6cc585393" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 33789 } health_report { overall_health: HEALTHY } } }
W20260809 15:04:28.054406 19554 tablet.cc:2369] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:28.073913 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.069191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.27.186.130:33789)
0809 15:04:28.069350 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:28.069354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:28.069373 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:04:28.069451 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:28.069455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:28.069529 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:28.069669 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:04:28.069951 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.070726 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.072008 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.072418 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.072525 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:28.072927 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:04:28.073376 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:04:28.073489 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:28.073536 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 15:04:28.073608 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:04:28.081394 19896 raft_consensus.cc:1261] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Refusing update from remote peer f35b6bc5b76144929d9f0fc6cc585393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:28.081588 19478 raft_consensus.cc:1261] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Refusing update from remote peer f35b6bc5b76144929d9f0fc6cc585393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:28.082173 20031 consensus_queue.cc:1035] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf94e38aed8d44468706a1020dc53bc5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:28.082616 20022 consensus_queue.cc:1035] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "921308cf39864144b8bed4446d8cf36a" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:28.110306 19263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:28.110591 19263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:28.113827 19263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Sending DeleteTablet for 3 replicas of tablet 54d26cdf4541455190397310ea96599c
I20260809 15:04:28.114995 19867 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d26cdf4541455190397310ea96599c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:28 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:43060
I20260809 15:04:28.115020 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d26cdf4541455190397310ea96599c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:28 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260809 15:04:28.115581 19651 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d26cdf4541455190397310ea96599c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:28 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:43020
I20260809 15:04:28.115664 20046 tablet_replica.cc:330] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: stopping tablet replica
I20260809 15:04:28.116091 20046 raft_consensus.cc:2229] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:28.116597 20046 raft_consensus.cc:2258] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:28.116740 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 19322
I20260809 15:04:28.118803 20046 ts_tablet_manager.cc:1905] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:28.126189 20046 ts_tablet_manager.cc:1918] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:28.126376 20048 tablet_replica.cc:330] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: stopping tablet replica
I20260809 15:04:28.126497 20046 log.cc:1196] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/54d26cdf4541455190397310ea96599c
I20260809 15:04:28.126770 20048 raft_consensus.cc:2229] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:04:28.127178 20046 ts_tablet_manager.cc:1939] T 54d26cdf4541455190397310ea96599c P bf94e38aed8d44468706a1020dc53bc5: Deleting consensus metadata
I20260809 15:04:28.127430 20048 raft_consensus.cc:2258] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:28.128968 19247 catalog_manager.cc:4841] TS bf94e38aed8d44468706a1020dc53bc5 (127.27.186.131:39085): tablet 54d26cdf4541455190397310ea96599c (table test-table [id=a1e9dc8da78e49ecb88f38673cc8c879]) successfully deleted
I20260809 15:04:28.129709 20048 ts_tablet_manager.cc:1905] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:28.130513 20047 tablet_replica.cc:330] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: stopping tablet replica
I20260809 15:04:28.130942 20047 raft_consensus.cc:2229] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:28.131453 20047 raft_consensus.cc:2258] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:28.133399 20047 ts_tablet_manager.cc:1905] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:28.138525 20048 ts_tablet_manager.cc:1918] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:28.138666 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.118144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.27.186.129:45865)
0809 15:04:28.118299 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:28.118304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:28.118323 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:04:28.118375 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:28.118380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:28.118463 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:28.118558 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:28.120561 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.121667 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.123902 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.124573 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.124691 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:28.130451 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:28.130465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:28.130481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:28.130525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:28.134643 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:28.135542 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:28.135550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:28.135553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:28.135731 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:28.136181 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:28.136186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:28.136190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:28.136582 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:28.136884 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:28.138089 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0809 15:04:28.138315 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:04:28.138983 20048 log.cc:1196] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/54d26cdf4541455190397310ea96599c
I20260809 15:04:28.139250 19545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:28.139675 20048 ts_tablet_manager.cc:1939] T 54d26cdf4541455190397310ea96599c P f35b6bc5b76144929d9f0fc6cc585393: Deleting consensus metadata
I20260809 15:04:28.141460 19245 catalog_manager.cc:4841] TS f35b6bc5b76144929d9f0fc6cc585393 (127.27.186.130:33789): tablet 54d26cdf4541455190397310ea96599c (table test-table [id=a1e9dc8da78e49ecb88f38673cc8c879]) successfully deleted
I20260809 15:04:28.281781 20047 ts_tablet_manager.cc:1918] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:28.282052 20047 log.cc:1196] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/54d26cdf4541455190397310ea96599c
I20260809 15:04:28.282539 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 19322
I20260809 15:04:28.282624 20047 ts_tablet_manager.cc:1939] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Deleting consensus metadata
W20260809 15:04:28.303491 19244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.186.129:45865 (error 108)
I20260809 15:04:28.303730 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 19557
I20260809 15:04:28.305953 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.305183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53734)
0809 15:04:28.305603 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.305727 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":193,"threads_started":1}
W20260809 15:04:28.306298 19244 catalog_manager.cc:4584] TS 921308cf39864144b8bed4446d8cf36a (127.27.186.129:45865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54d26cdf4541455190397310ea96599c: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
I20260809 15:04:28.317690 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.304603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.27.186.130:33789)
0809 15:04:28.304823 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:28.304829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:28.304852 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:04:28.304928 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:28.304933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:28.305023 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:28.305124 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:28.306812 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.307532 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.308869 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.309320 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.309428 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:28.311993 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:28.312005 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:28.312019 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:28.312061 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:28.314990 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:28.315638 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:28.315645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:28.315647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:28.315806 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:28.316165 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:28.316169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:28.316171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:28.316548 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:28.316868 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:28.317161 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 15:04:28.317385 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 15:04:28.318439 19741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:28.359831 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.359363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53746)
0809 15:04:28.359552 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.359656 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 15:04:28.360868 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.360457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53762)
0809 15:04:28.360609 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.360690 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 15:04:28.421341 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.420902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53768)
0809 15:04:28.421079 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.421198 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 15:04:28.422407 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.421986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53778)
0809 15:04:28.422159 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.422237 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 15:04:28.428608 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 19557
I20260809 15:04:28.455367 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 19754
I20260809 15:04:28.468124 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.456040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.27.186.131:39085)
0809 15:04:28.456210 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:28.456217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:28.456335 (+ 118us) server_negotiation.cc:407] Connection header received
0809 15:04:28.456517 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:28.456521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:28.456603 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:28.456687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:28.458055 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.458785 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.460072 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:28.460455 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:28.460567 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:28.463044 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:28.463058 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:28.463073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:28.463106 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:28.465479 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:28.466126 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:28.466135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:28.466138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:28.466285 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:28.466628 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:28.466632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:28.466634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:28.466996 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:28.467264 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:28.467603 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 15:04:28.467825 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:04:28.468710 19973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:28.503039 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.502569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53784)
0809 15:04:28.502769 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.502864 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 15:04:28.504019 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.503631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53792)
0809 15:04:28.503820 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.503891 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 15:04:28.555418 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 19754
I20260809 15:04:28.575609 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 19219
I20260809 15:04:28.576620 19310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:28.693951 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:28.693031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:45865 (local address 127.0.0.1:53808)
0809 15:04:28.693563 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:28.693727 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 15:04:28.694157 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 19219
I20260809 15:04:28.721580 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40839
--webserver_interface=127.27.186.190
--webserver_port=45309
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:29.031857 20061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:29.032104 20061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:29.032196 20061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:29.038993 20061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:29.039081 20061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:29.039132 20061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:29.039186 20061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:29.039229 20061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:29.048257 20061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=45309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:29.049284 20061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:29.050428 20061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:29.060793 20067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:29.060823 20068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:29.060854 20070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:29.061828 20061 server_base.cc:1034] running on GCE node
I20260809 15:04:29.062529 20061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:29.063728 20061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:29.064927 20061 hybrid_clock.cc:648] HybridClock initialized: now 1786287869064872 us; error 41 us; skew 500 ppm
I20260809 15:04:29.065228 20061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:29 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287869, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:29.068048 20061 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:29.069290 20061 webserver.cc:460] Webserver started at http://127.27.186.190:45309/ using document root <none> and password file <none>
I20260809 15:04:29.069864 20061 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:29.069962 20061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:29.074781 20061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:29.077296 20080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:29.078451 20061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:04:29.078572 20061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "51cf2597223245ff8a975aa72b946ccd"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "1bd4c417305110d654d0f1a28328d41b"
server_key_iv: "b4af4ebe527dcb77b30580891c6e2c73"
server_key_version: "encryptionkey@0"
I20260809 15:04:29.079391 20061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:29.089931 20061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:29.093228 20061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:29.093417 20061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:29.109685 20061 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:40839
I20260809 15:04:29.109707 20159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:40839 every 8 connection(s)
I20260809 15:04:29.110991 20061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:29.112731 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 20061
I20260809 15:04:29.113722 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:45865
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:40839
--webserver_port=39857
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:29.119177 20161 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:04:29.123274 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap starting.
I20260809 15:04:29.142292 20161 log.cc:824] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:29.157625 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:04:29.158157 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd: Bootstrap complete.
I20260809 15:04:29.166966 20161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:29.167757 20161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cf2597223245ff8a975aa72b946ccd, State: Initialized, Role: FOLLOWER
I20260809 15:04:29.168309 20161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:29.168534 20161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:29.168648 20161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:29.168815 20161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:04:29.172294 20161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:29.172746 20161 leader_election.cc:302] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cf2597223245ff8a975aa72b946ccd; no voters:
I20260809 15:04:29.173339 20161 leader_election.cc:288] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 15:04:29.173477 20168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:04:29.174504 20168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [term 3 LEADER]: Becoming Leader. State: Replica: 51cf2597223245ff8a975aa72b946ccd, State: Running, Role: LEADER
I20260809 15:04:29.175074 20168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } }
I20260809 15:04:29.175329 20161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:29.177609 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cf2597223245ff8a975aa72b946ccd. Latest consensus state: current_term: 3 leader_uuid: "51cf2597223245ff8a975aa72b946ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } } }
I20260809 15:04:29.178001 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:29.177637 20169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "51cf2597223245ff8a975aa72b946ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cf2597223245ff8a975aa72b946ccd" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40839 } } }
I20260809 15:04:29.178344 20169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cf2597223245ff8a975aa72b946ccd [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:29.191268 20178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:29.199980 20178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:29.201400 20178 catalog_manager.cc:1259] Loaded cluster ID: 4f4712e8f2ce475393f3682c5e4bcd9a
I20260809 15:04:29.201542 20178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 15:04:29.203644 20178 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f74407249ad google::LogMessage::Fail() at ??:0
@ 0x7f744072788a google::LogMessage::SendToLog() at ??:0
@ 0x7f74407244de google::LogMessage::Flush() at ??:0
@ 0x7f7440725a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7456157512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7456154976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f74561dc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f74561dc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7443370345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f744337391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7443373799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f744334eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f74466286db start_thread at ??:0
@ 0x7f743f5be61f clone at ??:0
W20260809 15:04:29.490437 20165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:29.490679 20165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:29.490797 20165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:29.497669 20165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:29.497785 20165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:29.497936 20165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:04:29.507078 20165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=39857
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:29.508150 20165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:29.509344 20165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:29.521364 20229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:29.521325 20234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:29.521423 20232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:29.522394 20165 server_base.cc:1034] running on GCE node
I20260809 15:04:29.522966 20165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:29.523787 20165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:29.524995 20165 hybrid_clock.cc:648] HybridClock initialized: now 1786287869524939 us; error 45 us; skew 500 ppm
I20260809 15:04:29.525286 20165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:29 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287869, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:29.528178 20165 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:04:29.529444 20165 webserver.cc:460] Webserver started at http://127.27.186.129:39857/ using document root <none> and password file <none>
I20260809 15:04:29.530092 20165 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:29.530215 20165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:29.535107 20165 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 15:04:29.537623 20242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:29.538944 20165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:04:29.539076 20165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "921308cf39864144b8bed4446d8cf36a"
format_stamp: "Formatted at 2026-08-09 15:04:23 on dist-test-slave-q9h9"
server_key: "0399b6f9db88c79851bf355b9111877d"
server_key_iv: "b0dbc7e1a8a32db9224a29e06c2e72b3"
server_key_version: "encryptionkey@0"
I20260809 15:04:29.539963 20165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:29.555424 20165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:29.558773 20165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:29.558991 20165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:29.559841 20165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:29.561394 20252 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260809 15:04:29.566380 20252 ts_tablet_manager.cc:1852] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260809 15:04:29.567322 20252 ts_tablet_manager.cc:1905] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:29.574002 20252 ts_tablet_manager.cc:1918] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:29.574275 20252 ts_tablet_manager.cc:1939] T 54d26cdf4541455190397310ea96599c P 921308cf39864144b8bed4446d8cf36a: Deleting consensus metadata
I20260809 15:04:29.574707 20165 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260809 15:04:29.574817 20165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260809 15:04:29.574947 20165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:29.575034 20165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:29.604703 20165 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:45865
I20260809 15:04:29.604784 20417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:45865 every 8 connection(s)
I20260809 15:04:29.605935 20165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:04:29.610369 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:29.608601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:34069)
0809 15:04:29.609296 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:29.609526 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":285,"threads_started":1}
W20260809 15:04:29.610882 20418 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:40839 (0 consecutive failures): Network error: Failed to ping master at 127.27.186.190:40839: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
I20260809 15:04:29.611378 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 20165
I20260809 15:04:29.611670 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:29.611365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:45081)
0809 15:04:29.611500 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:29.611573 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 15:04:29.611982 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:33789
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:40839
--webserver_port=46385
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:04:29.612511 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:29.612175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:47841)
0809 15:04:29.612309 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:29.612390 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 15:04:29.612736 20418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 15:04:29.933357 20429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:29.933584 20429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:29.933697 20429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:29.940644 20429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:29.940733 20429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:29.940886 20429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:04:29.950052 20429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:33789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=46385
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:29.951108 20429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:29.952286 20429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:29.964540 20439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:29.964581 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:29.964607 20440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:29.965503 20429 server_base.cc:1034] running on GCE node
I20260809 15:04:29.966002 20429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:29.966845 20429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:29.968042 20429 hybrid_clock.cc:648] HybridClock initialized: now 1786287869967993 us; error 43 us; skew 500 ppm
I20260809 15:04:29.968334 20429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:29 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287869, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:29.971239 20429 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:04:29.972471 20429 webserver.cc:460] Webserver started at http://127.27.186.130:46385/ using document root <none> and password file <none>
I20260809 15:04:29.973029 20429 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:29.973137 20429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:29.977998 20429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:29.980569 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:29.981750 20429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:04:29.981876 20429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "f35b6bc5b76144929d9f0fc6cc585393"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "ca0522b22d19a5f433eeea6e48b4774d"
server_key_iv: "817ec646be44412c81fed928cdc0050e"
server_key_version: "encryptionkey@0"
I20260809 15:04:29.982777 20429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:29.993934 20429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:29.997318 20429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:29.997532 20429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:29.998364 20429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:29.999634 20429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:29.999720 20429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:29.999795 20429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:29.999850 20429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:30.029347 20429 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:33789
I20260809 15:04:30.029409 20630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:33789 every 8 connection(s)
I20260809 15:04:30.030627 20429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:04:30.034881 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.033176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:33851)
0809 15:04:30.033835 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.034076 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":249,"threads_started":1}
W20260809 15:04:30.035502 20633 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:40839 (0 consecutive failures): Network error: Failed to ping master at 127.27.186.190:40839: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
I20260809 15:04:30.036190 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.035912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:43049)
0809 15:04:30.036032 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.036095 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 15:04:30.036887 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.036617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:45091)
0809 15:04:30.036717 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.036787 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 15:04:30.037127 20633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 15:04:30.039561 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 20429
I20260809 15:04:30.040126 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:39085
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:40839
--webserver_port=46239
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:30.355912 20637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:30.356151 20637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:30.356283 20637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:30.363178 20637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:30.363274 20637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:30.363425 20637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:04:30.372416 20637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:39085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=46239
--tserver_master_addrs=127.27.186.190:40839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:30.373553 20637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:30.374795 20637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:30.387032 20646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:30.387032 20647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:30.387286 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:30.387883 20637 server_base.cc:1034] running on GCE node
I20260809 15:04:30.388479 20637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:30.389312 20637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:30.390498 20637 hybrid_clock.cc:648] HybridClock initialized: now 1786287870390456 us; error 31 us; skew 500 ppm
I20260809 15:04:30.390775 20637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:30 dist-test-slave-q9h9 krb5kdc[18361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287870, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:30.393646 20637 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:04:30.394920 20637 webserver.cc:460] Webserver started at http://127.27.186.131:46239/ using document root <none> and password file <none>
I20260809 15:04:30.395525 20637 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:30.395624 20637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:30.400547 20637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:04:30.403021 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:30.404271 20637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:04:30.404412 20637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "bf94e38aed8d44468706a1020dc53bc5"
format_stamp: "Formatted at 2026-08-09 15:04:24 on dist-test-slave-q9h9"
server_key: "297b9578f4130c34cd5e90c59b5ae026"
server_key_iv: "d0250b28ba3ea1cc834cf58e494d39e7"
server_key_version: "encryptionkey@0"
I20260809 15:04:30.405296 20637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:30.421103 20637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:30.424535 20637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:30.424804 20637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:30.425827 20637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:30.427032 20637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:30.427136 20637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:30.427251 20637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:30.427323 20637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:30.460697 20637 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:39085
I20260809 15:04:30.460768 20844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:39085 every 8 connection(s)
I20260809 15:04:30.462057 20637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:04:30.464265 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 20637
I20260809 15:04:30.466843 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.464967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:60563)
0809 15:04:30.465763 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.465976 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":312,"threads_started":1}
W20260809 15:04:30.467490 20845 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:40839 (0 consecutive failures): Network error: Failed to ping master at 127.27.186.190:40839: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
I20260809 15:04:30.468182 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.467902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:53691)
0809 15:04:30.468019 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.468084 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 15:04:30.468904 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.468614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:56379)
0809 15:04:30.468729 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.468801 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 15:04:30.469125 20845 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 15:04:30.613929 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:30.613212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:54019)
0809 15:04:30.613631 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:30.613716 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":202,"threads_started":1}
I20260809 15:04:31.038437 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:31.037578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:38115)
0809 15:04:31.038073 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:31.038198 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":224,"threads_started":1}
I20260809 15:04:31.470391 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:31.469594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:50781)
0809 15:04:31.470011 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:31.470167 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":195,"threads_started":1}
I20260809 15:04:31.615507 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:31.614775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:47957)
0809 15:04:31.615240 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:31.615321 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260809 15:04:32.039949 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:32.039285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:48685)
0809 15:04:32.039702 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:32.039788 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260809 15:04:32.471863 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:32.471154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:38381)
0809 15:04:32.471602 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:32.471678 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260809 15:04:32.616968 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:32.616293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:35147)
0809 15:04:32.616726 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:32.616819 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260809 15:04:33.041292 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:33.040681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:42799)
0809 15:04:33.041067 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:33.041140 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":205,"threads_started":1}
I20260809 15:04:33.473268 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:33.472578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:38403)
0809 15:04:33.473018 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:33.473106 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
I20260809 15:04:33.618438 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:33.617739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:40767)
0809 15:04:33.618162 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:33.618242 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":192,"threads_started":1}
I20260809 15:04:34.042717 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:34.042004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:48639)
0809 15:04:34.042454 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:34.042532 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":200,"threads_started":1}
I20260809 15:04:34.474763 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:34.474083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:57439)
0809 15:04:34.474495 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:34.474573 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":195,"threads_started":1}
I20260809 15:04:34.619912 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:34.619198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:51467)
0809 15:04:34.619679 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:34.619767 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
I20260809 15:04:35.044112 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:35.043438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:43697)
0809 15:04:35.043862 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:35.043958 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260809 15:04:35.476227 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:35.475550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:40605)
0809 15:04:35.475978 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:35.476054 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":210,"threads_started":1}
I20260809 15:04:35.621286 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:35.620653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:51055)
0809 15:04:35.621049 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:35.621125 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":205,"threads_started":1}
I20260809 15:04:36.045529 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:36.044879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:44739)
0809 15:04:36.045281 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:36.045365 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":201,"threads_started":1}
I20260809 15:04:36.477643 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:36.476989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:60243)
0809 15:04:36.477390 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:36.477478 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":204,"threads_started":1}
I20260809 15:04:36.622902 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:36.622071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:49299)
0809 15:04:36.622625 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:36.622710 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
I20260809 15:04:37.047019 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:37.046305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:54623)
0809 15:04:37.046740 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:37.046819 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":193,"threads_started":1}
I20260809 15:04:37.479243 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:37.478487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:45133)
0809 15:04:37.478966 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:37.479041 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":225,"threads_started":1}
I20260809 15:04:37.624333 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:37.623636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:34023)
0809 15:04:37.624059 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:37.624159 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
I20260809 15:04:38.048540 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:38.047800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:40601)
0809 15:04:38.048224 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:38.048339 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260809 15:04:38.480789 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:38.480097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:38217)
0809 15:04:38.480544 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:38.480621 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":208,"threads_started":1}
I20260809 15:04:38.625913 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:38.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:37925)
0809 15:04:38.625619 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:38.625733 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":259,"threads_started":1}
I20260809 15:04:39.050215 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:39.049420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:55975)
0809 15:04:39.049930 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:39.050008 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260809 15:04:39.482225 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:39.481514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:45257)
0809 15:04:39.481957 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:39.482063 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":219,"threads_started":1}
I20260809 15:04:39.627571 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:39.626770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:56989)
0809 15:04:39.627243 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:39.627345 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
I20260809 15:04:40.051622 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:40.050970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:60001)
0809 15:04:40.051391 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:40.051468 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":224,"threads_started":1}
I20260809 15:04:40.483639 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:40.483013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:35883)
0809 15:04:40.483418 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:40.483495 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":215,"threads_started":1}
I20260809 15:04:40.629150 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:40.628455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:37775)
0809 15:04:40.628876 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:40.628955 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":234,"threads_started":1}
I20260809 15:04:41.053016 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:41.052382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:33083)
0809 15:04:41.052771 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:41.052853 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":189,"threads_started":1}
I20260809 15:04:41.485098 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:41.484397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:33901)
0809 15:04:41.484847 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:41.484934 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260809 15:04:41.630822 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:41.629996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:46769)
0809 15:04:41.630505 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:41.630608 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
I20260809 15:04:42.054459 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:42.053791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:42545)
0809 15:04:42.054229 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:42.054310 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":239,"threads_started":1}
I20260809 15:04:42.486583 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:42.485919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:52203)
0809 15:04:42.486335 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:42.486411 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":218,"threads_started":1}
I20260809 15:04:42.632328 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:42.631659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:52589)
0809 15:04:42.632061 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:42.632163 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":198,"threads_started":1}
I20260809 15:04:43.055958 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:43.055236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:56411)
0809 15:04:43.055678 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:43.055797 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":221,"threads_started":1}
I20260809 15:04:43.488050 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:43.487364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:34455)
0809 15:04:43.487782 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:43.487870 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":192,"threads_started":1}
I20260809 15:04:43.633747 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:43.633063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:52193)
0809 15:04:43.633483 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:43.633564 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":218,"threads_started":1}
I20260809 15:04:44.057366 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:44.056672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:54261)
0809 15:04:44.057124 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:44.057199 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260809 15:04:44.489485 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:44.488820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:54159)
0809 15:04:44.489241 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:44.489317 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260809 15:04:44.635298 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:44.634636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.129:37865)
0809 15:04:44.635051 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:44.635141 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
I20260809 15:04:45.059017 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:45.058336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.130:49505)
0809 15:04:45.058742 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:45.058851 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
I20260809 15:04:45.472862 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 20165
I20260809 15:04:45.490991 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:45.474238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.27.186.129:45865)
0809 15:04:45.474820 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:45.474852 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:45.474895 (+ 43us) server_negotiation.cc:407] Connection header received
0809 15:04:45.476923 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:45.476934 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:45.477351 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:45.477666 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:45.479149 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:45.480367 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:45.481566 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:45.482273 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:45.482447 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:45.485044 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:45.485052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:45.485065 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:45.485114 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:45.487903 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:45.488537 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:45.488545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:45.488550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:45.488708 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:45.489059 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:45.489062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:45.489065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:45.489563 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:45.489829 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:45.490333 (+ 504us) server_negotiation.cc:299] Negotiation successful
0809 15:04:45.490703 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":313,"threads_started":1}
I20260809 15:04:45.491307 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:45.490441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:40839 (local address 127.27.186.131:38083)
0809 15:04:45.490998 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:45.491106 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":351,"threads_started":1}
I20260809 15:04:45.492267 20412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:45.575034 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 20165
I20260809 15:04:45.598201 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 20429
I20260809 15:04:45.614668 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:45.598990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.27.186.130:33789)
0809 15:04:45.599471 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:45.599484 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:45.599520 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:04:45.600955 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:45.600964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:45.601366 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:45.601682 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:45.603178 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:45.604385 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:45.605588 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:45.606319 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:45.606521 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:45.608970 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:45.608978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:45.608990 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:45.609026 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:45.611593 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:45.612236 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:45.612243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:45.612249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:45.612422 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:45.612786 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:45.612790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:45.612792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:45.613256 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:45.613545 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:45.613955 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 15:04:45.614391 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260809 15:04:45.615895 20624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:45.697116 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 20429
I20260809 15:04:45.720124 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 20637
I20260809 15:04:45.736491 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:45.721048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.27.186.131:39085)
0809 15:04:45.721647 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:45.721660 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:45.721700 (+ 40us) server_negotiation.cc:407] Connection header received
0809 15:04:45.723425 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:45.723436 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:45.723858 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:45.724132 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:45.725336 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:45.726555 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:45.727930 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:45.728526 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:45.728658 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:45.731046 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:45.731054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:45.731066 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:45.731100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:45.733791 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:45.734438 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:45.734444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:45.734450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:45.734623 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:45.734938 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:45.734941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:45.734943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:45.735399 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:45.735622 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:45.735906 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 15:04:45.736206 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
I20260809 15:04:45.737722 20838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:45.821673 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 20637
2026-08-09T15:04:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24307 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:04:45 dist-test-slave-q9h9 krb5kdc[20934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:04:45 dist-test-slave-q9h9 krb5kdc[20934](info): set up 2 sockets
Aug 09 15:04:45 dist-test-slave-q9h9 krb5kdc[20934](info): commencing operation
krb5kdc: starting...
W20260809 15:04:47.025267 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:04:47 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:04:47.257892 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33419
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:47.572078 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:47.572347 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:47.572481 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:47.579350 20959 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 15:04:47.579440 20959 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 15:04:47.579502 20959 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 15:04:47.579576 20959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:04:47.579649 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:47.579742 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:47.579804 20959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:04:47.579878 20959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:04:47.588997 20959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:47.590058 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:47.591269 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:47.601933 20968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:47.601899 20971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:47.601977 20975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:47.602666 20959 server_base.cc:1034] running on GCE node
I20260809 15:04:47.603327 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:47.604558 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:47.605755 20959 hybrid_clock.cc:648] HybridClock initialized: now 1786287887605692 us; error 44 us; skew 500 ppm
I20260809 15:04:47.606094 20959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:47 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287887, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:47.608983 20959 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:04:47.610327 20959 webserver.cc:460] Webserver started at http://127.27.186.190:43417/ using document root <none> and password file <none>
I20260809 15:04:47.610936 20959 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:47.611023 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:47.611295 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:47.613984 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "d590dd100c874a2991e84c6498fd28d1"
format_stamp: "Formatted at 2026-08-09 15:04:47 on dist-test-slave-q9h9"
server_key: "0cabe154e3f151506c16e192382e9b82"
server_key_iv: "fe1cd6496f5069f36085bf38af9e82ec"
server_key_version: "encryptionkey@0"
I20260809 15:04:47.614805 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "d590dd100c874a2991e84c6498fd28d1"
format_stamp: "Formatted at 2026-08-09 15:04:47 on dist-test-slave-q9h9"
server_key: "0cabe154e3f151506c16e192382e9b82"
server_key_iv: "fe1cd6496f5069f36085bf38af9e82ec"
server_key_version: "encryptionkey@0"
I20260809 15:04:47.619479 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:04:47.622637 20986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:47.623790 20959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:04:47.623932 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "d590dd100c874a2991e84c6498fd28d1"
format_stamp: "Formatted at 2026-08-09 15:04:47 on dist-test-slave-q9h9"
server_key: "0cabe154e3f151506c16e192382e9b82"
server_key_iv: "fe1cd6496f5069f36085bf38af9e82ec"
server_key_version: "encryptionkey@0"
I20260809 15:04:47.624111 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:47.648806 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:47.652601 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:47.652845 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:47.669392 20959 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:33419
I20260809 15:04:47.669414 21066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:33419 every 8 connection(s)
I20260809 15:04:47.670562 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:04:47.673808 21067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:47.677872 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 20959
I20260809 15:04:47.678120 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:04:47.678993 28394 external_mini_cluster.cc:1428] Setting key 2681cb7ec9db7b7a463ccbb81204b1a8
I20260809 15:04:47.684578 21067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap starting.
I20260809 15:04:47.687817 21067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:47.688984 21067 log.cc:824] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:47.691995 21067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: No bootstrap required, opened a new log
Aug 09 15:04:47 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:47.701785 21067 raft_consensus.cc:348] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:04:47.702127 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:47.680859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.27.186.190:33419)
0809 15:04:47.681585 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:47.681596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:47.681638 (+ 42us) server_negotiation.cc:407] Connection header received
0809 15:04:47.683345 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:47.683367 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:47.683690 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:47.683973 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:47.685680 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:47.686981 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:47.688892 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:47.689676 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:47.689885 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:47.694244 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:47.694255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:47.694276 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:47.694325 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:47.698035 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:47.698668 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:47.698676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:47.698684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:47.698896 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:47.699222 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:47.699227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:47.699230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:47.699911 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:47.700195 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:47.700468 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 15:04:47.700863 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260809 15:04:47.702234 21067 raft_consensus.cc:374] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:47.702370 21067 raft_consensus.cc:729] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d590dd100c874a2991e84c6498fd28d1, State: Initialized, Role: FOLLOWER
I20260809 15:04:47.702943 21067 consensus_queue.cc:260] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:04:47.703166 21067 raft_consensus.cc:388] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:04:47.703343 21067 raft_consensus.cc:482] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:04:47.703536 21067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:47.705976 21067 raft_consensus.cc:504] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:04:47.706465 21067 leader_election.cc:302] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d590dd100c874a2991e84c6498fd28d1; no voters:
I20260809 15:04:47.707062 21067 leader_election.cc:288] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:04:47.707127 21074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:47.707785 21074 raft_consensus.cc:686] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 LEADER]: Becoming Leader. State: Replica: d590dd100c874a2991e84c6498fd28d1, State: Running, Role: LEADER
I20260809 15:04:47.708328 21074 consensus_queue.cc:237] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:04:47.708915 21067 sys_catalog.cc:564] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:04:47.710925 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d590dd100c874a2991e84c6498fd28d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } } }
I20260809 15:04:47.711314 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:47.712569 21076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d590dd100c874a2991e84c6498fd28d1. Latest consensus state: current_term: 1 leader_uuid: "d590dd100c874a2991e84c6498fd28d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } } }
I20260809 15:04:47.712850 21076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: This master's current role is: LEADER
I20260809 15:04:47.715067 21083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:04:47.720199 21083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:04:47.728207 21083 catalog_manager.cc:1347] Generated new cluster ID: a9b92d9ecabb4937b23c97ee2029888a
I20260809 15:04:47.728322 21083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:04:47.734777 21083 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:04:47.736366 21083 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:04:47.745724 21083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Generated new TSK 0
I20260809 15:04:47.746589 21083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:04:47.803118 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33419
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:48.121307 21106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:48.121584 21106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:48.121742 21106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:48.129513 21106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:48.129639 21106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:48.129803 21106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:04:48.139626 21106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:48.140738 21106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:48.142129 21106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:48.154708 21116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:48.154680 21119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:48.154809 21117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:48.155824 21106 server_base.cc:1034] running on GCE node
I20260809 15:04:48.156363 21106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:48.157223 21106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:48.158414 21106 hybrid_clock.cc:648] HybridClock initialized: now 1786287888158374 us; error 32 us; skew 500 ppm
I20260809 15:04:48.158686 21106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:48 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:48.161593 21106 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:04:48.162895 21106 webserver.cc:460] Webserver started at http://127.27.186.129:36095/ using document root <none> and password file <none>
I20260809 15:04:48.163450 21106 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:48.163548 21106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:48.163793 21106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:48.166451 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "98763aaeb6a74c2684384b8f96711499"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "ee06d532ab543bcbe5b0ae23c50c2231"
server_key_iv: "15c8668738837451346b745e19b85b54"
server_key_version: "encryptionkey@0"
I20260809 15:04:48.167128 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "98763aaeb6a74c2684384b8f96711499"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "ee06d532ab543bcbe5b0ae23c50c2231"
server_key_iv: "15c8668738837451346b745e19b85b54"
server_key_version: "encryptionkey@0"
I20260809 15:04:48.171878 21106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:04:48.175063 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:48.176234 21106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:04:48.176366 21106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "98763aaeb6a74c2684384b8f96711499"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "ee06d532ab543bcbe5b0ae23c50c2231"
server_key_iv: "15c8668738837451346b745e19b85b54"
server_key_version: "encryptionkey@0"
I20260809 15:04:48.176527 21106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:48.221166 21106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:48.225032 21106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:48.225288 21106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:48.226150 21106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:48.227382 21106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:48.227452 21106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:48.227564 21106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:48.227618 21106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:48.259133 21106 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:43033
I20260809 15:04:48.259195 21307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:43033 every 8 connection(s)
I20260809 15:04:48.260375 21106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:04:48.263722 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 21106
I20260809 15:04:48.263912 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:04:48.264585 28394 external_mini_cluster.cc:1428] Setting key c42cff18817e11e1cf9a8409ef26081b
Aug 09 15:04:48 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:48.280565 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:48.262786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:39439 (local address 127.27.186.190:33419)
0809 15:04:48.263226 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:48.263231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:48.264132 (+ 901us) server_negotiation.cc:407] Connection header received
0809 15:04:48.266143 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:48.266148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:48.266236 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:48.266352 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:48.268292 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.269023 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.271074 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.271487 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.271585 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:48.275252 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:48.275257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:48.275261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:48.275292 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:48.277612 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:48.278345 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:48.278349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:48.278352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:48.278475 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:48.278889 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:48.278892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:48.278894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:48.279140 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:48.279305 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:48.279973 (+ 668us) server_negotiation.cc:299] Negotiation successful
0809 15:04:48.280197 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260809 15:04:48.281368 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:48.263176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:39439)
0809 15:04:48.263957 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:48.264011 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:48.265879 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:48.266624 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:48.266634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:48.267359 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:48.268111 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:48.268138 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.269188 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.269193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:48.270947 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:48.270963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.271715 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.271722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:48.271979 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:48.272634 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:48.272662 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:48.275049 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:48.277754 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:48.277763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:48.277777 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:48.278223 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:48.278582 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:48.278587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:48.278589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:48.278773 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:48.279268 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:48.279276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:48.279605 (+ 329us) client_negotiation.cc:769] Sending connection context
0809 15:04:48.279820 (+ 215us) client_negotiation.cc:240] Negotiation successful
0809 15:04:48.280288 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":318,"threads_started":1}
I20260809 15:04:48.282823 21309 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33419
I20260809 15:04:48.283116 21309 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:48.283950 21309 heartbeater.cc:511] Master 127.27.186.190:33419 requested a full tablet report, sending...
I20260809 15:04:48.286213 21017 ts_manager.cc:194] Registered new tserver with Master: 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033)
I20260809 15:04:48.288105 21017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:39439
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:04:48.322774 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33419
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:48.635649 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:48.635890 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:48.635979 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:48.642776 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:48.642866 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:48.642985 21315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:04:48.651993 21315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:48.653030 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:48.654206 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:48.666460 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:48.666502 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:48.666524 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:48.667119 21315 server_base.cc:1034] running on GCE node
I20260809 15:04:48.667630 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:48.668475 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:48.669692 21315 hybrid_clock.cc:648] HybridClock initialized: now 1786287888669649 us; error 30 us; skew 500 ppm
I20260809 15:04:48.670068 21315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:48 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:48.673619 21315 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:04:48.675016 21315 webserver.cc:460] Webserver started at http://127.27.186.130:42989/ using document root <none> and password file <none>
I20260809 15:04:48.675633 21315 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:48.675745 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:48.676026 21315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:48.678808 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "5f491a9798f74674a07d0b3c45aca433"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "41aa2818fb315b307aa4892c1212a2b7"
server_key_iv: "ae9e1ea6fcd85986900ffc151bbcd532"
server_key_version: "encryptionkey@0"
I20260809 15:04:48.679548 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "5f491a9798f74674a07d0b3c45aca433"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "41aa2818fb315b307aa4892c1212a2b7"
server_key_iv: "ae9e1ea6fcd85986900ffc151bbcd532"
server_key_version: "encryptionkey@0"
I20260809 15:04:48.684580 21315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260809 15:04:48.687844 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:48.688978 21315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 15:04:48.689121 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "5f491a9798f74674a07d0b3c45aca433"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "41aa2818fb315b307aa4892c1212a2b7"
server_key_iv: "ae9e1ea6fcd85986900ffc151bbcd532"
server_key_version: "encryptionkey@0"
I20260809 15:04:48.689312 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:48.706720 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:48.710248 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:48.710500 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:48.711344 21315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:48.712579 21315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:48.712652 21315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:48.712767 21315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:48.712818 21315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:48.740783 21315 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:38873
I20260809 15:04:48.740840 21530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:38873 every 8 connection(s)
I20260809 15:04:48.741910 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:04:48.752038 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 21315
I20260809 15:04:48.752233 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:04:48.752718 28394 external_mini_cluster.cc:1428] Setting key 6b800232d11b711a508ea3063838889d
Aug 09 15:04:48 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:48.760278 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:48.744203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:45543 (local address 127.27.186.190:33419)
0809 15:04:48.744374 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:48.744378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:48.745381 (+ 1003us) server_negotiation.cc:407] Connection header received
0809 15:04:48.747156 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:48.747162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:48.747242 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:48.747330 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:48.748729 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.749444 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.750980 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.751400 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.751512 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:48.755017 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:48.755022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:48.755025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:48.755055 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:48.757422 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:48.758263 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:48.758267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:48.758269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:48.758391 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:48.758826 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:48.758829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:48.758831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:48.759075 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:48.759183 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:48.759891 (+ 708us) server_negotiation.cc:299] Negotiation successful
0809 15:04:48.760050 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:04:48.761479 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:48.744556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:45543)
0809 15:04:48.745233 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:48.745269 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:48.746895 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:48.747561 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:48.747569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:48.748030 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:48.748553 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:48.748571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.749600 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.749604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:48.750833 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:48.750847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:48.751620 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:48.751628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:48.751843 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:48.752486 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:48.752509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:48.754790 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:48.757560 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:48.757568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:48.757580 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:48.758162 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:48.758487 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:48.758491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:48.758494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:48.758734 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:48.759190 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:48.759198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:48.759549 (+ 351us) client_negotiation.cc:769] Sending connection context
0809 15:04:48.759828 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 15:04:48.760276 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260809 15:04:48.762857 21531 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33419
I20260809 15:04:48.763116 21531 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:48.763777 21531 heartbeater.cc:511] Master 127.27.186.190:33419 requested a full tablet report, sending...
I20260809 15:04:48.765081 21017 ts_manager.cc:194] Registered new tserver with Master: 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:04:48.765827 21017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:45543
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:04:48.806603 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33419
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:04:49.116043 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:04:49.116297 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:04:49.116433 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:04:49.123297 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:04:49.123404 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:04:49.123541 21538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:04:49.133071 21538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:04:49.134186 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:04:49.135423 21538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:04:49.147495 21547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:49.147540 21546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:04:49.147550 21549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:04:49.148352 21538 server_base.cc:1034] running on GCE node
I20260809 15:04:49.148892 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:04:49.149808 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:04:49.151021 21538 hybrid_clock.cc:648] HybridClock initialized: now 1786287889150949 us; error 68 us; skew 500 ppm
I20260809 15:04:49.151324 21538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:04:49 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287889, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:04:49.159327 21538 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:04:49.160687 21538 webserver.cc:460] Webserver started at http://127.27.186.131:41017/ using document root <none> and password file <none>
I20260809 15:04:49.161314 21538 fs_manager.cc:359] Metadata directory not provided
I20260809 15:04:49.161414 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:04:49.161667 21538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:04:49.164474 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2"
format_stamp: "Formatted at 2026-08-09 15:04:49 on dist-test-slave-q9h9"
server_key: "eafa08188c8b238dfd347631c53d75f1"
server_key_iv: "27d9541cd09478a72cbb1b3cad1860a5"
server_key_version: "encryptionkey@0"
I20260809 15:04:49.165187 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2"
format_stamp: "Formatted at 2026-08-09 15:04:49 on dist-test-slave-q9h9"
server_key: "eafa08188c8b238dfd347631c53d75f1"
server_key_iv: "27d9541cd09478a72cbb1b3cad1860a5"
server_key_version: "encryptionkey@0"
I20260809 15:04:49.170089 21538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260809 15:04:49.173293 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:49.174525 21538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 15:04:49.174654 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2"
format_stamp: "Formatted at 2026-08-09 15:04:49 on dist-test-slave-q9h9"
server_key: "eafa08188c8b238dfd347631c53d75f1"
server_key_iv: "27d9541cd09478a72cbb1b3cad1860a5"
server_key_version: "encryptionkey@0"
I20260809 15:04:49.174816 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:04:49.184473 21538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:04:49.188228 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:04:49.188450 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:04:49.189253 21538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:04:49.190441 21538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:04:49.190513 21538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:49.190598 21538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:04:49.190640 21538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:04:49.220355 21538 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:34517
I20260809 15:04:49.220409 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:34517 every 8 connection(s)
I20260809 15:04:49.221481 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:04:49.225834 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 21538
I20260809 15:04:49.226033 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:04:49.226467 28394 external_mini_cluster.cc:1428] Setting key c0d02232a6a109a7d71e5c1bef175fdb
Aug 09 15:04:49 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287889, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:04:49.239583 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.223762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:42405 (local address 127.27.186.190:33419)
0809 15:04:49.223921 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.223926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.224938 (+ 1012us) server_negotiation.cc:407] Connection header received
0809 15:04:49.226838 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.226844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.226927 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.227032 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:49.228895 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.229783 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.231243 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.231632 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.231728 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.234719 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:49.234724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:49.234728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:49.234758 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:49.237161 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:49.237764 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:49.237767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:49.237769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:49.237880 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:49.238273 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:49.238277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:49.238279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:49.238553 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:49.238658 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.239202 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.239350 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:04:49.240583 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.224144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:42405)
0809 15:04:49.224795 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.224832 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.226529 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.227304 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.227315 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.228038 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:04:49.228730 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.228753 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.229921 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.229925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.231130 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.231146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.231807 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.231811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.231996 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.232515 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:04:49.232535 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:04:49.234538 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:04:49.237280 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:49.237286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:49.237296 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:49.237673 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:49.237955 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:04:49.237958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:04:49.237960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:04:49.238176 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:04:49.238649 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:04:49.238654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:04:49.238916 (+ 262us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.239116 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.239453 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":253,"threads_started":1}
I20260809 15:04:49.241824 21743 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33419
I20260809 15:04:49.242121 21743 heartbeater.cc:464] Registering TS with master...
I20260809 15:04:49.242746 21743 heartbeater.cc:511] Master 127.27.186.190:33419 requested a full tablet report, sending...
I20260809 15:04:49.244024 21015 ts_manager.cc:194] Registered new tserver with Master: 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517)
I20260809 15:04:49.244859 21015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:42405
I20260809 15:04:49.254068 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:04:49.271498 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.258590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.27.186.190:33419)
0809 15:04:49.258774 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.258779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.259323 (+ 544us) server_negotiation.cc:407] Connection header received
0809 15:04:49.259450 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.259455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.259533 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.259640 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:49.261027 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.262164 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.264138 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.264558 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.264659 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.267073 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:49.267077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:49.267081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:49.267109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:49.269363 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:49.269986 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:49.269989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:49.269991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:49.270149 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:49.270436 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:49.270439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:49.270441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:49.270747 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:49.270932 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.271137 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.271290 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 15:04:49.275137 21015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:04:49.282572 21015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:04:49.291388 21309 heartbeater.cc:503] Master 127.27.186.190:33419 was elected leader, sending a full tablet report...
I20260809 15:04:49.317267 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.294498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38873 (local address 127.0.0.1:56124)
0809 15:04:49.302521 (+ 8023us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.302540 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.302658 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.310430 (+ 7772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.310437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.310476 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.310793 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.310805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.312471 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.312476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.314868 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.314883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.316292 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.316298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.316499 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.316561 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.316698 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.316887 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7864,"mutex_wait_us":578,"thread_start_us":7131,"threads_started":1}
I20260809 15:04:49.317538 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.293775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34517 (local address 127.0.0.1:43184)
0809 15:04:49.294751 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.294790 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.294959 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.310867 (+ 15908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.310871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.310897 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.311410 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.311423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.313523 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.313528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.315255 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.315269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.316660 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.316666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.316842 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.316874 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.316962 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.317092 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":59,"thread_start_us":611,"threads_started":1}
I20260809 15:04:49.317866 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.294083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.27.186.131:34517)
0809 15:04:49.310349 (+ 16266us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.310357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.310385 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:04:49.310476 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.310482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.310678 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.310859 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.311619 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.313394 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.315412 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.316469 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.316644 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.316808 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.317368 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.317534 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16112,"thread_start_us":15924,"threads_started":1}
I20260809 15:04:49.318425 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.294878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.27.186.130:38873)
0809 15:04:49.309752 (+ 14874us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.309765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.309798 (+ 33us) server_negotiation.cc:407] Connection header received
0809 15:04:49.309908 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.309917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.310232 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.310411 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.310989 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.312311 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.315045 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.316090 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.316242 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.316413 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.318049 (+ 1636us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.318183 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14486,"thread_start_us":14674,"threads_started":1}
I20260809 15:04:49.319538 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.295154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:35588)
0809 15:04:49.305468 (+ 10314us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.305484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.305579 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.310820 (+ 5241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.310825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.310855 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.311353 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.311366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.313190 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.313195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.315273 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.315284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.318759 (+ 3475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.318765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.318963 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.318997 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.319063 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.319184 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10205,"mutex_wait_us":20,"thread_start_us":10487,"threads_started":1}
I20260809 15:04:49.320516 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.295406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.27.186.129:43033)
0809 15:04:49.310003 (+ 14597us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.310032 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.310061 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:04:49.310152 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.310158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.310374 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.310554 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.311514 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.313051 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.316894 (+ 3843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.318397 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.318577 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.319978 (+ 1401us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.320043 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.320207 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14432,"thread_start_us":14263,"threads_started":1}
I20260809 15:04:49.321491 21644 tablet_service.cc:1467] Processing CreateTablet for tablet b033677f77fb48b28f98a0101a97baa1 (DEFAULT_TABLE table=test-table [id=df43e61437de4ccc91a577601e6751b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:49.321882 21422 tablet_service.cc:1467] Processing CreateTablet for tablet b033677f77fb48b28f98a0101a97baa1 (DEFAULT_TABLE table=test-table [id=df43e61437de4ccc91a577601e6751b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:49.322788 21644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b033677f77fb48b28f98a0101a97baa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:49.323261 21422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b033677f77fb48b28f98a0101a97baa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:49.323792 21214 tablet_service.cc:1467] Processing CreateTablet for tablet b033677f77fb48b28f98a0101a97baa1 (DEFAULT_TABLE table=test-table [id=df43e61437de4ccc91a577601e6751b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:04:49.325098 21214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b033677f77fb48b28f98a0101a97baa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:04:49.333719 21772 tablet_bootstrap.cc:492] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Bootstrap starting.
I20260809 15:04:49.336774 21772 tablet_bootstrap.cc:654] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:49.337007 21774 tablet_bootstrap.cc:492] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Bootstrap starting.
I20260809 15:04:49.338142 21772 log.cc:824] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:49.338505 21775 tablet_bootstrap.cc:492] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Bootstrap starting.
I20260809 15:04:49.340636 21774 tablet_bootstrap.cc:654] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:49.341482 21775 tablet_bootstrap.cc:654] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Neither blocks nor log segments found. Creating new log.
I20260809 15:04:49.342281 21772 tablet_bootstrap.cc:492] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: No bootstrap required, opened a new log
I20260809 15:04:49.342425 21775 log.cc:824] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:49.342545 21772 ts_tablet_manager.cc:1397] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 15:04:49.342573 21774 log.cc:824] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Log is configured to *not* fsync() on all Append() calls
I20260809 15:04:49.344659 21775 tablet_bootstrap.cc:492] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: No bootstrap required, opened a new log
I20260809 15:04:49.344898 21775 ts_tablet_manager.cc:1397] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260809 15:04:49.348412 21774 tablet_bootstrap.cc:492] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: No bootstrap required, opened a new log
I20260809 15:04:49.348657 21774 ts_tablet_manager.cc:1397] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260809 15:04:49.352144 21775 raft_consensus.cc:348] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.352463 21775 raft_consensus.cc:374] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:49.352559 21775 raft_consensus.cc:729] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98763aaeb6a74c2684384b8f96711499, State: Initialized, Role: FOLLOWER
I20260809 15:04:49.352849 21772 raft_consensus.cc:348] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.353245 21772 raft_consensus.cc:374] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:49.353098 21775 consensus_queue.cc:260] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.353344 21772 raft_consensus.cc:729] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa887bb4a7b4f918534ab9d3eb0b7c2, State: Initialized, Role: FOLLOWER
I20260809 15:04:49.353891 21772 consensus_queue.cc:260] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.354751 21775 ts_tablet_manager.cc:1428] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260809 15:04:49.359028 21774 raft_consensus.cc:348] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.359508 21774 raft_consensus.cc:374] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:04:49.359616 21774 raft_consensus.cc:729] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f491a9798f74674a07d0b3c45aca433, State: Initialized, Role: FOLLOWER
I20260809 15:04:49.360210 21774 consensus_queue.cc:260] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.361186 21531 heartbeater.cc:503] Master 127.27.186.190:33419 was elected leader, sending a full tablet report...
I20260809 15:04:49.361331 21743 heartbeater.cc:503] Master 127.27.186.190:33419 was elected leader, sending a full tablet report...
I20260809 15:04:49.361944 21772 ts_tablet_manager.cc:1428] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260809 15:04:49.362252 21774 ts_tablet_manager.cc:1428] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260809 15:04:49.472986 21745 tablet.cc:2369] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:04:49.493649 21532 tablet.cc:2369] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:04:49.512632 21310 tablet.cc:2369] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:04:49.734207 21785 raft_consensus.cc:482] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:49.734499 21785 raft_consensus.cc:504] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.736197 21785 leader_election.cc:288] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:04:49.738932 21787 raft_consensus.cc:482] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:04:49.739241 21787 raft_consensus.cc:504] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.740810 21787 leader_election.cc:288] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:04:49.749100 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.741147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.27.186.131:58863)
0809 15:04:49.741460 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.741478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.741580 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.742503 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.742509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.742537 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.742799 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.742809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.744243 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.744249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.746321 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.746337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.748300 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.748306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.748538 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.748575 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.748663 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.748810 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260809 15:04:49.749747 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.741930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:58863 (local address 127.27.186.129:43033)
0809 15:04:49.742131 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.742136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.742154 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:04:49.742206 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.742210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.742275 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.742354 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.742959 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.744063 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.746722 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.747987 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.748136 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.748211 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.749356 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.749493 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:04:49.750631 21245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b033677f77fb48b28f98a0101a97baa1" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98763aaeb6a74c2684384b8f96711499" is_pre_election: true
I20260809 15:04:49.751412 21245 raft_consensus.cc:2454] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 0.
I20260809 15:04:49.752187 21566 leader_election.cc:302] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa887bb4a7b4f918534ab9d3eb0b7c2, 98763aaeb6a74c2684384b8f96711499; no voters:
I20260809 15:04:49.752583 21787 raft_consensus.cc:2781] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:04:49.752722 21787 raft_consensus.cc:482] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:04:49.752835 21787 raft_consensus.cc:3037] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:49.756994 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.736467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:33715 (local address 127.27.186.131:34517)
0809 15:04:49.736655 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.736660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.737501 (+ 841us) server_negotiation.cc:407] Connection header received
0809 15:04:49.737685 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.737689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.737751 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.737825 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.738427 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.739322 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.754580 (+ 15258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.755910 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.756052 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.756117 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.756636 (+ 519us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.756764 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260809 15:04:49.757638 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.736552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34517 (local address 127.27.186.129:33715)
0809 15:04:49.737384 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.737404 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.737535 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.737862 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.737866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.737891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.738284 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.738296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.739446 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.739451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.754390 (+ 14939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.754406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.756303 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.756309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.756475 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.756506 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.757141 (+ 635us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.757283 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":28,"thread_start_us":427,"threads_started":1}
I20260809 15:04:49.758252 21787 raft_consensus.cc:504] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.758589 21681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b033677f77fb48b28f98a0101a97baa1" candidate_uuid: "98763aaeb6a74c2684384b8f96711499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" is_pre_election: true
I20260809 15:04:49.759083 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.741270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38873 (local address 127.27.186.131:40097)
0809 15:04:49.750407 (+ 9137us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.750424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.750522 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.751411 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.751416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.751440 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.751629 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.751639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.754211 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.754217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.756172 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.756187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.758214 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.758221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.758469 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.758501 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.758565 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.758687 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9039,"mutex_wait_us":1,"thread_start_us":10759,"threads_started":1}
I20260809 15:04:49.759130 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.736752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38873 (local address 127.27.186.129:39435)
0809 15:04:49.737489 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0809 15:04:49.737506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:04:49.737637 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:04:49.737933 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:04:49.737937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:04:49.737964 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:04:49.738377 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.738386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.739653 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.739658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.753073 (+ 13415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:04:49.753089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.758446 (+ 5357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.758453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:04:49.758610 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.758642 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:04:49.758719 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 15:04:49.758849 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":438,"threads_started":1}
I20260809 15:04:49.759090 21681 raft_consensus.cc:2379] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98763aaeb6a74c2684384b8f96711499 in current term 1: Already voted for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in this term.
I20260809 15:04:49.759091 21787 leader_election.cc:288] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 election: Requested vote from peers 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:04:49.759850 21245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b033677f77fb48b28f98a0101a97baa1" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98763aaeb6a74c2684384b8f96711499"
I20260809 15:04:49.759945 21799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.741357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:40097 (local address 127.27.186.130:38873)
0809 15:04:49.751118 (+ 9761us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.751123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.751147 (+ 24us) server_negotiation.cc:407] Connection header received
0809 15:04:49.751201 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.751205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.751269 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.752463 (+ 1194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.752773 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.754054 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.756644 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.757970 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.759372 (+ 1402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.759440 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.759491 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.759615 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":9640,"thread_start_us":9467,"threads_started":1}
I20260809 15:04:49.760046 21245 raft_consensus.cc:3037] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:49.760418 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.736808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:39435 (local address 127.27.186.130:38873)
0809 15:04:49.737009 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.737014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.737589 (+ 575us) server_negotiation.cc:407] Connection header received
0809 15:04:49.737768 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.737772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.737832 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.737906 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:04:49.738537 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.739487 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.756055 (+ 16568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.758114 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.758234 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.758303 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.759663 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.759803 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":747,"server-negotiator.queue_time_us":67}
I20260809 15:04:49.760856 21455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b033677f77fb48b28f98a0101a97baa1" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f491a9798f74674a07d0b3c45aca433"
I20260809 15:04:49.761283 21455 raft_consensus.cc:3037] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:04:49.761157 21457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b033677f77fb48b28f98a0101a97baa1" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f491a9798f74674a07d0b3c45aca433" is_pre_election: true
I20260809 15:04:49.761682 21454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b033677f77fb48b28f98a0101a97baa1" candidate_uuid: "98763aaeb6a74c2684384b8f96711499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f491a9798f74674a07d0b3c45aca433" is_pre_election: true
I20260809 15:04:49.762650 21140 leader_election.cc:302] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98763aaeb6a74c2684384b8f96711499; no voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2
I20260809 15:04:49.763059 21245 raft_consensus.cc:2454] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 1.
I20260809 15:04:49.763221 21785 raft_consensus.cc:2726] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:04:49.763447 21566 leader_election.cc:302] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa887bb4a7b4f918534ab9d3eb0b7c2, 98763aaeb6a74c2684384b8f96711499; no voters:
I20260809 15:04:49.763708 21787 raft_consensus.cc:2781] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:04:49.764122 21787 raft_consensus.cc:686] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa887bb4a7b4f918534ab9d3eb0b7c2, State: Running, Role: LEADER
I20260809 15:04:49.764671 21455 raft_consensus.cc:2454] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 1.
I20260809 15:04:49.764601 21787 consensus_queue.cc:237] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:04:49.769080 21015 catalog_manager.cc:5495] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } health_report { overall_health: UNKNOWN } } }
I20260809 15:04:49.882447 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.876691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.27.186.131:34517)
0809 15:04:49.876857 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.876862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.877278 (+ 416us) server_negotiation.cc:407] Connection header received
0809 15:04:49.877476 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.877480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.877598 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.877730 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:04:49.878194 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.878978 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.880503 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.880948 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.881040 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.881476 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:04:49.881950 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:04:49.882073 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.882150 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.882214 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:04:49.889428 21455 raft_consensus.cc:1261] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Refusing update from remote peer 6fa887bb4a7b4f918534ab9d3eb0b7c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:49.889509 21245 raft_consensus.cc:1261] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Refusing update from remote peer 6fa887bb4a7b4f918534ab9d3eb0b7c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:04:49.890210 21801 consensus_queue.cc:1035] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:49.890525 21787 consensus_queue.cc:1035] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:04:49.903812 21813 mvcc.cc:205] Tried to move back new op lower bound from 7316635196978720768 to 7316635196479688704. Current Snapshot: MvccSnapshot[applied={T|T < 7316635196978720768}]
I20260809 15:04:49.913283 21015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:49.913484 21015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:04:49.916486 21015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Sending DeleteTablet for 3 replicas of tablet b033677f77fb48b28f98a0101a97baa1
I20260809 15:04:49.917476 21644 tablet_service.cc:1514] Processing DeleteTablet for tablet b033677f77fb48b28f98a0101a97baa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:49 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:43184
I20260809 15:04:49.917706 21422 tablet_service.cc:1514] Processing DeleteTablet for tablet b033677f77fb48b28f98a0101a97baa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:49 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:56124
I20260809 15:04:49.917846 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 21106
I20260809 15:04:49.918545 21214 tablet_service.cc:1514] Processing DeleteTablet for tablet b033677f77fb48b28f98a0101a97baa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:04:49 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:35588
I20260809 15:04:49.931017 21829 tablet_replica.cc:330] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: stopping tablet replica
I20260809 15:04:49.931422 21829 raft_consensus.cc:2229] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:49.931864 21829 raft_consensus.cc:2258] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:49.932641 21825 tablet_replica.cc:330] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: stopping tablet replica
I20260809 15:04:49.932730 21826 tablet_replica.cc:330] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: stopping tablet replica
I20260809 15:04:49.933014 21825 raft_consensus.cc:2229] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:04:49.933169 21826 raft_consensus.cc:2229] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:04:49.933621 21826 raft_consensus.cc:2258] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:49.933908 21829 ts_tablet_manager.cc:1905] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:49.935338 21826 ts_tablet_manager.cc:1905] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:49.940915 21829 ts_tablet_manager.cc:1918] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:49.940915 21825 raft_consensus.cc:2258] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:04:49.941160 21829 log.cc:1196] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/b033677f77fb48b28f98a0101a97baa1
Aug 09 15:04:49 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:04:49.941743 21829 ts_tablet_manager.cc:1939] T b033677f77fb48b28f98a0101a97baa1 P 98763aaeb6a74c2684384b8f96711499: Deleting consensus metadata
I20260809 15:04:49.941928 21826 ts_tablet_manager.cc:1918] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:49.942142 21826 log.cc:1196] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/b033677f77fb48b28f98a0101a97baa1
I20260809 15:04:49.942628 21825 ts_tablet_manager.cc:1905] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:04:49.942694 21826 ts_tablet_manager.cc:1939] T b033677f77fb48b28f98a0101a97baa1 P 5f491a9798f74674a07d0b3c45aca433: Deleting consensus metadata
I20260809 15:04:49.943683 20992 catalog_manager.cc:4841] TS 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033): tablet b033677f77fb48b28f98a0101a97baa1 (table test-table [id=df43e61437de4ccc91a577601e6751b6]) successfully deleted
I20260809 15:04:49.944793 20988 catalog_manager.cc:4841] TS 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): tablet b033677f77fb48b28f98a0101a97baa1 (table test-table [id=df43e61437de4ccc91a577601e6751b6]) successfully deleted
I20260809 15:04:49.948288 21825 ts_tablet_manager.cc:1918] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:04:49.948438 21825 log.cc:1196] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/b033677f77fb48b28f98a0101a97baa1
I20260809 15:04:49.948868 21825 ts_tablet_manager.cc:1939] T b033677f77fb48b28f98a0101a97baa1 P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Deleting consensus metadata
I20260809 15:04:49.950081 20992 catalog_manager.cc:4841] TS 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): tablet b033677f77fb48b28f98a0101a97baa1 (table test-table [id=df43e61437de4ccc91a577601e6751b6]) successfully deleted
I20260809 15:04:49.950836 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:49.918967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.27.186.129:43033)
0809 15:04:49.919145 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:49.919151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:49.930842 (+ 11691us) server_negotiation.cc:407] Connection header received
0809 15:04:49.931125 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:49.931130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:49.931205 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:49.931283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:49.933318 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.934400 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.936277 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:49.936921 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:49.937012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:49.942813 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:49.942828 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:49.942843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:49.942886 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:49.946744 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:49.947567 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:49.947576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:49.947580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:49.947812 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:49.948308 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:49.948313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:49.948317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:49.948775 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:49.949062 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:49.950254 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0809 15:04:49.950463 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:04:49.951424 21302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:50.047070 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 21106
I20260809 15:04:50.076972 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 21315
Aug 09 15:04:50 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:04:50.093003 21799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:50.077759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.27.186.130:38873)
0809 15:04:50.077951 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:50.077956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:50.078183 (+ 227us) server_negotiation.cc:407] Connection header received
0809 15:04:50.078395 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:50.078401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:50.078499 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:50.078619 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:50.079898 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:50.080988 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:50.082238 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:50.082831 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:50.082954 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:50.087541 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:50.087554 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:50.087568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:50.087601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:50.090275 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:50.091076 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:50.091083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:50.091085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:50.091203 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:50.091553 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:50.091556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:50.091558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:50.091873 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:50.092118 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:50.092498 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 15:04:50.092678 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 15:04:50.093658 21524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:50.182771 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 21315
I20260809 15:04:50.205436 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 21538
Aug 09 15:04:50 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:04:50.224992 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:04:50.206183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.27.186.131:34517)
0809 15:04:50.206371 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:04:50.206376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:04:50.206533 (+ 157us) server_negotiation.cc:407] Connection header received
0809 15:04:50.206749 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:04:50.206754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:04:50.206826 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:04:50.206912 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:04:50.208764 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:50.209539 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:50.211585 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:04:50.212051 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:04:50.212158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:04:50.217398 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:04:50.217417 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:04:50.217440 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:04:50.217490 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:04:50.221528 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:50.222475 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:50.222485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:50.222489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:50.222769 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:04:50.223190 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:04:50.223200 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:04:50.223203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:04:50.223714 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:04:50.224030 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0809 15:04:50.224390 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 15:04:50.224635 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:04:50.225741 21737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:50.342888 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 21538
I20260809 15:04:50.367008 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 20959
I20260809 15:04:50.368214 21061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:04:50.473534 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 20959
I20260809 15:05:10.504763 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33419
--webserver_interface=127.27.186.190
--webserver_port=43417
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:10.821979 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:10.822245 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:10.822367 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:10.829121 21835 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 15:05:10.829205 21835 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 15:05:10.829277 21835 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 15:05:10.829350 21835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:05:10.829416 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:10.829511 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:10.829568 21835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:05:10.829643 21835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:05:10.838785 21835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=43417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:10.839823 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:10.841022 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:10.862174 21847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:10.862210 21848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:10.862249 21851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:10.862731 21835 server_base.cc:1034] running on GCE node
I20260809 15:05:10.863667 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:10.865103 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:10.866321 21835 hybrid_clock.cc:648] HybridClock initialized: now 1786287910866254 us; error 33 us; skew 500 ppm
I20260809 15:05:10.866677 21835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:10 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287910, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:10.874749 21835 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:05:10.876308 21835 webserver.cc:460] Webserver started at http://127.27.186.190:43417/ using document root <none> and password file <none>
I20260809 15:05:10.877046 21835 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:10.877167 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:10.882845 21835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:05:10.885573 21862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:10.886858 21835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:05:10.887019 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "d590dd100c874a2991e84c6498fd28d1"
format_stamp: "Formatted at 2026-08-09 15:04:47 on dist-test-slave-q9h9"
server_key: "0cabe154e3f151506c16e192382e9b82"
server_key_iv: "fe1cd6496f5069f36085bf38af9e82ec"
server_key_version: "encryptionkey@0"
I20260809 15:05:10.887980 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:10.920121 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:10.924104 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:10.924371 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:10.941308 21835 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:33419
I20260809 15:05:10.941326 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:33419 every 8 connection(s)
I20260809 15:05:10.942679 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:05:10.944931 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 21835
I20260809 15:05:10.945717 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:43033
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:33419
--webserver_port=36095
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:05:10.949247 21946 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:05:10.952418 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap starting.
I20260809 15:05:10.975843 21946 log.cc:824] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:10.990875 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:05:10.991510 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap complete.
I20260809 15:05:11.004112 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:11.005211 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d590dd100c874a2991e84c6498fd28d1, State: Initialized, Role: FOLLOWER
I20260809 15:05:11.005913 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:11.006223 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:05:11.006356 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:05:11.006527 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:05:11.011857 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:11.012488 21946 leader_election.cc:302] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d590dd100c874a2991e84c6498fd28d1; no voters:
I20260809 15:05:11.013080 21946 leader_election.cc:288] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 15:05:11.013196 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:05:11.014303 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 LEADER]: Becoming Leader. State: Replica: d590dd100c874a2991e84c6498fd28d1, State: Running, Role: LEADER
I20260809 15:05:11.014837 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:11.015189 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:05:11.017156 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d590dd100c874a2991e84c6498fd28d1. Latest consensus state: current_term: 2 leader_uuid: "d590dd100c874a2991e84c6498fd28d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } } }
I20260809 15:05:11.017536 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:11.018033 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d590dd100c874a2991e84c6498fd28d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } } }
I20260809 15:05:11.018349 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:11.020147 21964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:05:11.027333 21964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:05:11.028707 21964 catalog_manager.cc:1259] Loaded cluster ID: a9b92d9ecabb4937b23c97ee2029888a
I20260809 15:05:11.028855 21964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:05:11.032876 21964 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:05:11.034354 21964 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Loaded TSK: 0
I20260809 15:05:11.068743 21978 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 15:05:11.073132 21964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Generated new TSK 1
I20260809 15:05:11.073488 21964 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 15:05:11.308663 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:11.308916 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:11.309031 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:11.316051 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:11.316167 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:11.316363 21949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:05:11.325693 21949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:43033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=36095
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:11.326874 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:11.328078 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:11.341995 21991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:11.342167 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:11.342981 21994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:11.343004 21949 server_base.cc:1034] running on GCE node
I20260809 15:05:11.343667 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:11.344540 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:11.345736 21949 hybrid_clock.cc:648] HybridClock initialized: now 1786287911345672 us; error 57 us; skew 500 ppm
I20260809 15:05:11.346045 21949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:11 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287911, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:11.349136 21949 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:05:11.350575 21949 webserver.cc:460] Webserver started at http://127.27.186.129:36095/ using document root <none> and password file <none>
I20260809 15:05:11.351217 21949 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:11.351351 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:11.356537 21949 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:05:11.359160 22008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:11.360436 21949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:05:11.360575 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "98763aaeb6a74c2684384b8f96711499"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "ee06d532ab543bcbe5b0ae23c50c2231"
server_key_iv: "15c8668738837451346b745e19b85b54"
server_key_version: "encryptionkey@0"
I20260809 15:05:11.361496 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:11.376937 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:11.380589 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:11.380816 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:11.381700 21949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:11.383068 21949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:11.383158 21949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:11.383265 21949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:11.383344 21949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:11.413533 21949 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:43033
I20260809 15:05:11.413599 22198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:43033 every 8 connection(s)
I20260809 15:05:11.414897 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:05:11.418175 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 21949
I20260809 15:05:11.418967 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:38873
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:33419
--webserver_port=42989
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:05:11 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287911, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:11.438445 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:11.417472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:39505 (local address 127.27.186.190:33419)
0809 15:05:11.418133 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:11.418149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:11.418732 (+ 583us) server_negotiation.cc:407] Connection header received
0809 15:05:11.421382 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:11.421395 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:11.421689 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:11.421993 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:11.423735 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.424928 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.426964 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.427609 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.427734 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:11.431603 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:11.431611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:11.431623 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:11.431657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:11.434474 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:11.435273 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:11.435279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:11.435284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:11.435439 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:11.435917 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:11.435920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:11.435922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:11.436321 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:11.436469 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:11.437267 (+ 798us) server_negotiation.cc:299] Negotiation successful
0809 15:05:11.437552 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":244,"threads_started":1}
I20260809 15:05:11.438732 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:11.417604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:39505)
0809 15:05:11.418530 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:11.418591 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:11.420881 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:11.422189 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:11.422199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:11.422854 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:11.423542 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:11.423566 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.425095 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.425100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:11.426839 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:11.426854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.427828 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.427836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:11.428116 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:11.428758 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:11.428796 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:11.431335 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:11.434609 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:11.434617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:11.434629 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:11.435158 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:11.435551 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:11.435555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:11.435559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:11.435823 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:11.436451 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:11.436457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:11.436798 (+ 341us) client_negotiation.cc:769] Sending connection context
0809 15:05:11.437108 (+ 310us) client_negotiation.cc:240] Negotiation successful
0809 15:05:11.437581 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":423,"threads_started":1}
I20260809 15:05:11.441326 22199 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33419
I20260809 15:05:11.441588 22199 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:11.442298 22199 heartbeater.cc:511] Master 127.27.186.190:33419 requested a full tablet report, sending...
I20260809 15:05:11.444268 21887 ts_manager.cc:194] Registered new tserver with Master: 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033)
I20260809 15:05:11.445706 21887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:39505
W20260809 15:05:11.743849 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:11.744113 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:11.744244 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:11.751181 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:11.751287 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:11.751426 22209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:05:11.760608 22209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:38873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=42989
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:11.761680 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:11.762902 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:11.775132 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:11.775070 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:11.775166 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:11.775930 22209 server_base.cc:1034] running on GCE node
I20260809 15:05:11.776466 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:11.777307 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:11.778497 22209 hybrid_clock.cc:648] HybridClock initialized: now 1786287911778452 us; error 36 us; skew 500 ppm
I20260809 15:05:11.778780 22209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:11 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287911, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:11.781620 22209 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:05:11.782900 22209 webserver.cc:460] Webserver started at http://127.27.186.130:42989/ using document root <none> and password file <none>
I20260809 15:05:11.783495 22209 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:11.783624 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:11.788436 22209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:05:11.790902 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:11.792155 22209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:05:11.792307 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "5f491a9798f74674a07d0b3c45aca433"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "41aa2818fb315b307aa4892c1212a2b7"
server_key_iv: "ae9e1ea6fcd85986900ffc151bbcd532"
server_key_version: "encryptionkey@0"
I20260809 15:05:11.793221 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:11.832950 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:11.837069 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:11.837327 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:11.838320 22209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:11.839639 22209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:11.839742 22209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:11.839851 22209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:11.839922 22209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:11.869926 22209 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:38873
I20260809 15:05:11.869990 22426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:38873 every 8 connection(s)
I20260809 15:05:11.871353 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:05:11.877995 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 22209
I20260809 15:05:11.878621 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:34517
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:33419
--webserver_port=41017
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 15:05:11 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287911, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:11.889739 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:11.873694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:38387 (local address 127.27.186.190:33419)
0809 15:05:11.873870 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:11.873875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:11.875013 (+ 1138us) server_negotiation.cc:407] Connection header received
0809 15:05:11.876773 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:11.876779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:11.876859 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:11.876946 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:11.878465 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.879180 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.880656 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.881066 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.881173 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:11.884450 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:11.884455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:11.884458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:11.884491 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:11.886930 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:11.887599 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:11.887603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:11.887605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:11.887723 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:11.888124 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:11.888127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:11.888129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:11.888403 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:11.888536 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:11.889179 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 15:05:11.889418 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:05:11.890607 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:11.874159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:38387)
0809 15:05:11.874859 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:11.874903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:11.876513 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:11.877183 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:11.877190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:11.877686 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:11.878304 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:11.878325 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.879336 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.879340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:11.880515 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:11.880528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:11.881278 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:11.881286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:11.881498 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:11.882108 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:11.882130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:11.884240 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:11.887068 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:11.887077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:11.887090 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:11.887476 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:11.887829 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:11.887834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:11.887838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:11.888011 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:11.888526 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:11.888533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:11.888829 (+ 296us) client_negotiation.cc:769] Sending connection context
0809 15:05:11.889054 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 15:05:11.889509 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":253,"threads_started":1}
I20260809 15:05:11.891901 22428 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33419
I20260809 15:05:11.892226 22428 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:11.893031 22428 heartbeater.cc:511] Master 127.27.186.190:33419 requested a full tablet report, sending...
I20260809 15:05:11.894558 21887 ts_manager.cc:194] Registered new tserver with Master: 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:05:11.895571 21887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:38387
W20260809 15:05:12.200151 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:12.200392 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:12.200487 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:12.207545 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:12.207640 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:12.207765 22435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:05:12.216909 22435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:34517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=41017
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:12.217941 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:12.219136 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:12.231029 22443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:12.231067 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:12.231040 22444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:12.231719 22435 server_base.cc:1034] running on GCE node
I20260809 15:05:12.232280 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:12.233100 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:12.234288 22435 hybrid_clock.cc:648] HybridClock initialized: now 1786287912234237 us; error 45 us; skew 500 ppm
I20260809 15:05:12.234563 22435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:12 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287912, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:12.237411 22435 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:05:12.238732 22435 webserver.cc:460] Webserver started at http://127.27.186.131:41017/ using document root <none> and password file <none>
I20260809 15:05:12.239322 22435 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:12.239446 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:12.244270 22435 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:05:12.246810 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:12.247987 22435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:05:12.248123 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2"
format_stamp: "Formatted at 2026-08-09 15:04:49 on dist-test-slave-q9h9"
server_key: "eafa08188c8b238dfd347631c53d75f1"
server_key_iv: "27d9541cd09478a72cbb1b3cad1860a5"
server_key_version: "encryptionkey@0"
I20260809 15:05:12.249018 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:12.265467 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:12.269248 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:12.269491 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:12.270397 22435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:12.271642 22435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:12.271730 22435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:12.271832 22435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:12.271898 22435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:12.301674 22435 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:34517
I20260809 15:05:12.301771 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:34517 every 8 connection(s)
I20260809 15:05:12.303146 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:05:12.306313 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 22435
Aug 09 15:05:12 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287912, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:12.321738 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.307361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.27.186.190:33419)
0809 15:05:12.308131 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.308135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.308149 (+ 14us) server_negotiation.cc:407] Connection header received
0809 15:05:12.308198 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.308202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.308260 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.308349 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:12.310481 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.311244 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.312608 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.313010 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.313152 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.316352 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:12.316357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:12.316360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:12.316389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:12.319121 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.319873 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.319878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.319880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.319998 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.320364 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.320369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.320372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.320628 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:12.320870 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.321150 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.321359 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":690,"thread_start_us":589,"threads_started":1}
I20260809 15:05:12.323100 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.305502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:58035 (local address 127.27.186.190:33419)
0809 15:05:12.305726 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.305732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.306874 (+ 1142us) server_negotiation.cc:407] Connection header received
0809 15:05:12.309060 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.309063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.309103 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.309156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:12.310883 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.311650 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.313103 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.313562 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.313688 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.317492 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:12.317497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:12.317500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:12.317541 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:12.320143 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.320825 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.320829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.320831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.320940 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.321363 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.321367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.321369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.321664 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:12.321810 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.322585 (+ 775us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.322796 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 15:05:12.324254 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.305898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:58035)
0809 15:05:12.306704 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.306754 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:12.308812 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:12.309390 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:12.309401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:12.310075 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:12.310726 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.310744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.311801 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.311804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.312969 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.312981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.313769 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.313774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.314107 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.314659 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:12.314690 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:12.317282 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:12.320279 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:12.320288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:12.320302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:12.320697 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:12.321043 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:12.321048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:12.321052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:12.321245 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:12.321810 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:12.321817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:12.322149 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 15:05:12.322479 (+ 330us) client_negotiation.cc:240] Negotiation successful
0809 15:05:12.322893 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":314,"threads_started":1}
I20260809 15:05:12.325443 22634 heartbeater.cc:347] Connected to a master server at 127.27.186.190:33419
I20260809 15:05:12.325732 22634 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:12.326419 22634 heartbeater.cc:511] Master 127.27.186.190:33419 requested a full tablet report, sending...
I20260809 15:05:12.327668 21887 ts_manager.cc:194] Registered new tserver with Master: 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517)
I20260809 15:05:12.328433 21887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:58035
I20260809 15:05:12.336458 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:05:12.342185 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.337242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.27.186.190:33419)
0809 15:05:12.337411 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.337417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.337776 (+ 359us) server_negotiation.cc:407] Connection header received
0809 15:05:12.337926 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.337929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.338068 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.338205 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:05:12.338547 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.339253 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.340574 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.340970 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.341057 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.341471 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:05:12.341701 (+ 230us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 15:05:12.341939 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34520: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 15:05:12.342304 22203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34520: token expired
W20260809 15:05:12.342789 21747 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.186.190:33419)
I20260809 15:05:12.354163 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.343199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.27.186.190:33419)
0809 15:05:12.343338 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.343341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.343355 (+ 14us) server_negotiation.cc:407] Connection header received
0809 15:05:12.343467 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.343470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.343515 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.343582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:12.344827 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.345497 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.346809 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.347212 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.347305 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.349598 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:12.349603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:12.349607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:12.349636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:12.351960 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.352583 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.352586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.352588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.352694 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.352986 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.352989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.352991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.353275 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:12.353480 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.353737 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.353875 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:05:12.357040 21887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:05:12.364035 21887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:05:12.386052 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.374969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.27.186.131:34517)
0809 15:05:12.375560 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.375568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.376292 (+ 724us) server_negotiation.cc:407] Connection header received
0809 15:05:12.376683 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.376692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.376951 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.377108 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:12.378187 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.379968 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.382904 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.384419 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.384582 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.384765 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.385488 (+ 723us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.385672 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":283,"threads_started":1}
I20260809 15:05:12.386813 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.374812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34517 (local address 127.0.0.1:40870)
0809 15:05:12.376142 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.376185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:12.376448 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:12.377477 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:12.377485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:12.377518 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:12.377952 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.377964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.380160 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.380166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.382691 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.382708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.385017 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.385024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.385228 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.385276 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 15:05:12.386255 (+ 979us) client_negotiation.cc:240] Negotiation successful
0809 15:05:12.386459 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":902,"threads_started":1}
I20260809 15:05:12.390846 22533 tablet_service.cc:1467] Processing CreateTablet for tablet 0de4826bcccc4c65885544abecc90b6c (DEFAULT_TABLE table=test-table [id=6a2c552750404b91be11d80204a89582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:12.392437 22533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de4826bcccc4c65885544abecc90b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:12.395560 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.375413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38873 (local address 127.0.0.1:36266)
0809 15:05:12.386627 (+ 11214us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.386645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:12.386752 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:12.388465 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:12.388470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:12.388494 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:12.388899 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.388912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.390780 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.390785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.392676 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.392690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.394843 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.394850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.395029 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.395063 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:05:12.395149 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 15:05:12.395260 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11089,"threads_started":1}
I20260809 15:05:12.396241 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.375921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.27.186.130:38873)
0809 15:05:12.387606 (+ 11685us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.387614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.387646 (+ 32us) server_negotiation.cc:407] Connection header received
0809 15:05:12.387762 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.387769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.387980 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.388173 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:12.389113 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.390582 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.392866 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.394319 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.394464 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.394611 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.395765 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.395917 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11525,"threads_started":1}
I20260809 15:05:12.399861 22324 tablet_service.cc:1467] Processing CreateTablet for tablet 0de4826bcccc4c65885544abecc90b6c (DEFAULT_TABLE table=test-table [id=6a2c552750404b91be11d80204a89582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:12.401005 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.376478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33350)
0809 15:05:12.387068 (+ 10590us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.387084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:12.387165 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:12.393836 (+ 6671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:12.393842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:12.393871 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:12.394081 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.394092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.396861 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.396866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.398649 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.398664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.400357 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.400363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.400530 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.400561 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:05:12.400645 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 15:05:12.400778 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10500,"thread_start_us":9904,"threads_started":1}
I20260809 15:05:12.401196 22324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de4826bcccc4c65885544abecc90b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:12.401456 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.376961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.27.186.129:43033)
0809 15:05:12.393042 (+ 16081us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.393050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.393081 (+ 31us) server_negotiation.cc:407] Connection header received
0809 15:05:12.393176 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.393184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.393433 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.394266 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:12.394492 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.396734 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.398822 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.400185 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.400348 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.400564 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.400722 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.400871 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15894,"thread_start_us":16615,"threads_started":1}
I20260809 15:05:12.404709 22089 tablet_service.cc:1467] Processing CreateTablet for tablet 0de4826bcccc4c65885544abecc90b6c (DEFAULT_TABLE table=test-table [id=6a2c552750404b91be11d80204a89582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:12.405712 22656 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Bootstrap starting.
I20260809 15:05:12.406565 22089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de4826bcccc4c65885544abecc90b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:12.408567 22656 tablet_bootstrap.cc:654] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:12.409601 22656 log.cc:824] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:12.412315 22656 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: No bootstrap required, opened a new log
I20260809 15:05:12.412554 22656 ts_tablet_manager.cc:1397] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260809 15:05:12.422147 22661 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Bootstrap starting.
I20260809 15:05:12.422609 22656 raft_consensus.cc:348] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.423004 22656 raft_consensus.cc:374] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:12.423107 22656 raft_consensus.cc:729] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa887bb4a7b4f918534ab9d3eb0b7c2, State: Initialized, Role: FOLLOWER
I20260809 15:05:12.423857 22656 consensus_queue.cc:260] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.426726 22661 tablet_bootstrap.cc:654] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:12.427480 22662 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Bootstrap starting.
I20260809 15:05:12.428084 22661 log.cc:824] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:12.430626 22634 heartbeater.cc:503] Master 127.27.186.190:33419 was elected leader, sending a full tablet report...
I20260809 15:05:12.431041 22662 tablet_bootstrap.cc:654] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:12.431237 22656 ts_tablet_manager.cc:1428] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260809 15:05:12.432098 22661 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: No bootstrap required, opened a new log
I20260809 15:05:12.432209 22662 log.cc:824] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:12.432374 22661 ts_tablet_manager.cc:1397] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 15:05:12.435087 22662 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: No bootstrap required, opened a new log
I20260809 15:05:12.435325 22662 ts_tablet_manager.cc:1397] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 15:05:12.441992 22661 raft_consensus.cc:348] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.442421 22661 raft_consensus.cc:374] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:12.442524 22661 raft_consensus.cc:729] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f491a9798f74674a07d0b3c45aca433, State: Initialized, Role: FOLLOWER
I20260809 15:05:12.443054 22661 consensus_queue.cc:260] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.444010 22428 heartbeater.cc:503] Master 127.27.186.190:33419 was elected leader, sending a full tablet report...
I20260809 15:05:12.444602 22661 ts_tablet_manager.cc:1428] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Time spent starting tablet: real 0.012s user 0.002s sys 0.010s
I20260809 15:05:12.445153 22662 raft_consensus.cc:348] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.445456 22662 raft_consensus.cc:374] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:12.445539 22662 raft_consensus.cc:729] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98763aaeb6a74c2684384b8f96711499, State: Initialized, Role: FOLLOWER
I20260809 15:05:12.445994 22662 consensus_queue.cc:260] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.446867 22199 heartbeater.cc:503] Master 127.27.186.190:33419 was elected leader, sending a full tablet report...
I20260809 15:05:12.447454 22662 ts_tablet_manager.cc:1428] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260809 15:05:12.554694 22635 tablet.cc:2369] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:05:12.596292 22663 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:12.596516 22663 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.598094 22663 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:05:12.604036 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.598529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:45253 (local address 127.27.186.130:38873)
0809 15:05:12.598935 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.598941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.598958 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:05:12.599022 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.599026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.599080 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.599153 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:12.599582 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.600487 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.602296 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.603276 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.603385 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.603437 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.603753 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.603836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260809 15:05:12.604161 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.598546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38873 (local address 127.27.186.131:45253)
0809 15:05:12.598721 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.598742 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:12.598860 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:12.599212 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:12.599216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:12.599243 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:12.599441 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.599454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.600613 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.600619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.602137 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.602158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.603416 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.603422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.603605 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.603641 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 15:05:12.603740 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 15:05:12.603870 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 15:05:12.604827 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.598569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:34063 (local address 127.27.186.129:43033)
0809 15:05:12.598761 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.598767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.599367 (+ 600us) server_negotiation.cc:407] Connection header received
0809 15:05:12.599488 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.599492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.599565 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.599634 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:12.600210 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.601083 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.602944 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.603893 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.604029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.604092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.604456 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.604578 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 15:05:12.604879 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.598546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.27.186.131:34063)
0809 15:05:12.599269 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.599283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:12.599390 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:12.599670 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:12.599675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:12.599700 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:12.600071 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.600083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.601218 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.601222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.602796 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:12.602810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.604099 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.604105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:12.604278 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.604310 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:05:12.604407 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 15:05:12.604520 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":42,"thread_start_us":379,"threads_started":1}
I20260809 15:05:12.605165 22362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de4826bcccc4c65885544abecc90b6c" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f491a9798f74674a07d0b3c45aca433" is_pre_election: true
I20260809 15:05:12.605664 22362 raft_consensus.cc:2454] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 0.
I20260809 15:05:12.605860 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de4826bcccc4c65885544abecc90b6c" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98763aaeb6a74c2684384b8f96711499" is_pre_election: true
I20260809 15:05:12.606303 22463 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2; no voters:
I20260809 15:05:12.606329 22120 raft_consensus.cc:2454] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 0.
I20260809 15:05:12.606611 22663 raft_consensus.cc:2781] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:05:12.606751 22663 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:05:12.606868 22663 raft_consensus.cc:3037] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:12.608919 22663 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.609467 22663 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 election: Requested vote from peers 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
I20260809 15:05:12.609808 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de4826bcccc4c65885544abecc90b6c" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98763aaeb6a74c2684384b8f96711499"
I20260809 15:05:12.609910 22362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de4826bcccc4c65885544abecc90b6c" candidate_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f491a9798f74674a07d0b3c45aca433"
I20260809 15:05:12.609967 22120 raft_consensus.cc:3037] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:12.610112 22362 raft_consensus.cc:3037] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:12.612106 22120 raft_consensus.cc:2454] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 1.
I20260809 15:05:12.612218 22362 raft_consensus.cc:2454] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa887bb4a7b4f918534ab9d3eb0b7c2 in term 1.
I20260809 15:05:12.612486 22464 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa887bb4a7b4f918534ab9d3eb0b7c2, 98763aaeb6a74c2684384b8f96711499; no voters:
I20260809 15:05:12.612742 22663 raft_consensus.cc:2781] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:05:12.613137 22663 raft_consensus.cc:686] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa887bb4a7b4f918534ab9d3eb0b7c2, State: Running, Role: LEADER
I20260809 15:05:12.613611 22663 consensus_queue.cc:237] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:12.617774 21887 catalog_manager.cc:5495] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131). New cstate: current_term: 1 leader_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } health_report { overall_health: UNKNOWN } } }
W20260809 15:05:12.623062 22431 tablet.cc:2369] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:05:12.647465 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.642877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.27.186.131:34517)
0809 15:05:12.643031 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.643036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.643110 (+ 74us) server_negotiation.cc:407] Connection header received
0809 15:05:12.643262 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.643266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.643331 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.643442 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:05:12.643731 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.644394 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.645666 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.646108 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.646230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.646646 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:05:12.647055 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:05:12.647187 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.647228 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.647288 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:05:12.654598 22362 raft_consensus.cc:1261] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Refusing update from remote peer 6fa887bb4a7b4f918534ab9d3eb0b7c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:05:12.654637 22120 raft_consensus.cc:1261] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Refusing update from remote peer 6fa887bb4a7b4f918534ab9d3eb0b7c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:05:12.655491 22671 consensus_queue.cc:1035] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:05:12.655848 22663 consensus_queue.cc:1035] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:05:12.668886 22201 tablet.cc:2369] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:05:12.679134 21886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:05:12.679381 21886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:05:12.682516 21886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Sending DeleteTablet for 3 replicas of tablet 0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:12.683480 22533 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de4826bcccc4c65885544abecc90b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:12 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:40870
I20260809 15:05:12.683609 22089 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de4826bcccc4c65885544abecc90b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:12 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:33350
I20260809 15:05:12.683975 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de4826bcccc4c65885544abecc90b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:12 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:36266
I20260809 15:05:12.685024 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 21949
I20260809 15:05:12.693222 22692 tablet_replica.cc:330] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: stopping tablet replica
I20260809 15:05:12.694141 22692 raft_consensus.cc:2229] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:05:12.694737 22692 raft_consensus.cc:2258] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:12.696905 22692 ts_tablet_manager.cc:1905] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:12.697968 22690 tablet_replica.cc:330] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: stopping tablet replica
I20260809 15:05:12.703636 22692 ts_tablet_manager.cc:1918] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:12.704236 22691 tablet_replica.cc:330] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: stopping tablet replica
I20260809 15:05:12.704586 22691 raft_consensus.cc:2229] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:05:12.704782 22690 raft_consensus.cc:2229] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:05:12.705072 22691 raft_consensus.cc:2258] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:12.705099 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.686476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.27.186.129:43033)
0809 15:05:12.686643 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.686648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.686665 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:05:12.686727 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.686732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.686815 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.686903 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:12.688825 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.690077 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.692395 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.693059 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.693175 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.697099 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:12.697111 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:12.697124 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:12.697160 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:12.700795 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.701639 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.701647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.701651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.701814 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.702296 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.702301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.702304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.702803 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:12.703067 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.704287 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.704524 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 15:05:12.705471 22692 log.cc:1196] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:12.706040 22692 ts_tablet_manager.cc:1939] T 0de4826bcccc4c65885544abecc90b6c P 5f491a9798f74674a07d0b3c45aca433: Deleting consensus metadata
I20260809 15:05:12.706205 22188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:12.707219 22690 raft_consensus.cc:2258] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:12.707691 21866 catalog_manager.cc:4841] TS 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): tablet 0de4826bcccc4c65885544abecc90b6c (table test-table [id=6a2c552750404b91be11d80204a89582]) successfully deleted
I20260809 15:05:12.709362 22690 ts_tablet_manager.cc:1905] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:12.717154 22690 ts_tablet_manager.cc:1918] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:12.717348 22690 log.cc:1196] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:12.717865 22690 ts_tablet_manager.cc:1939] T 0de4826bcccc4c65885544abecc90b6c P 6fa887bb4a7b4f918534ab9d3eb0b7c2: Deleting consensus metadata
I20260809 15:05:12.719480 21867 catalog_manager.cc:4841] TS 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): tablet 0de4826bcccc4c65885544abecc90b6c (table test-table [id=6a2c552750404b91be11d80204a89582]) successfully deleted
I20260809 15:05:12.707149 22691 ts_tablet_manager.cc:1905] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:12.797082 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 21949
I20260809 15:05:12.827482 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 22209
W20260809 15:05:12.827531 21867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.186.129:43033 (error 108)
I20260809 15:05:12.830829 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.830350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33368)
0809 15:05:12.830541 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.830656 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 15:05:12.831103 21867 catalog_manager.cc:4584] TS 98763aaeb6a74c2684384b8f96711499 (127.27.186.129:43033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0de4826bcccc4c65885544abecc90b6c: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
I20260809 15:05:12.842430 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.828302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.27.186.130:38873)
0809 15:05:12.828488 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.828495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.828807 (+ 312us) server_negotiation.cc:407] Connection header received
0809 15:05:12.828885 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.828891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.828961 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.829048 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:12.830565 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.831570 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.833370 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.833756 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.833845 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.837036 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:12.837050 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:12.837064 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:12.837102 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:12.839814 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.840633 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.840638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.840641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.840766 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.841121 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.841123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.841125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.841453 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:12.841662 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.841986 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.842177 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:05:12.843101 22420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:12.884565 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.884096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33378)
0809 15:05:12.884310 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.884415 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 15:05:12.885699 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.885247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33392)
0809 15:05:12.885407 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.885507 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 15:05:12.929493 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 22209
I20260809 15:05:12.946372 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.945823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33402)
0809 15:05:12.946107 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.946203 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260809 15:05:12.947475 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.947026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33406)
0809 15:05:12.947244 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:12.947325 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260809 15:05:12.961030 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 22435
I20260809 15:05:12.976238 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:12.961736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.27.186.131:34517)
0809 15:05:12.961942 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:12.961948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:12.961971 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:05:12.962118 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:12.962124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:12.962236 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:12.962364 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:12.963732 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.964891 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.966203 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:12.966880 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:12.967006 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:12.969495 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:12.969510 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:12.969534 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:12.969587 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:12.973407 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.974064 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.974074 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.974077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.974292 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:12.974638 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:12.974643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:12.974646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:12.975105 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:12.975445 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:12.975604 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 15:05:12.975832 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 15:05:12.976962 22623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:13.027961 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:13.027528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33408)
0809 15:05:13.027723 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:13.027826 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 15:05:13.029000 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:13.028610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33416)
0809 15:05:13.028789 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:13.028864 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 15:05:13.096037 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 22435
I20260809 15:05:13.121110 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 21835
I20260809 15:05:13.122269 21940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:13.230654 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:13.230148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43033 (local address 127.0.0.1:33432)
0809 15:05:13.230341 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:13.230466 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.129:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 15:05:13.230739 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 21835
I20260809 15:05:33.256816 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33419
--webserver_interface=127.27.186.190
--webserver_port=43417
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:33.577090 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:33.577322 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:33.577432 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:33.584262 22699 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 15:05:33.584345 22699 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 15:05:33.584398 22699 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 15:05:33.584465 22699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:05:33.584527 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:33.584626 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:33.584679 22699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:05:33.584753 22699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:05:33.593807 22699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=43417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:33.594849 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:33.596017 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:33.606698 22709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:33.606742 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:33.606775 22710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:33.607523 22699 server_base.cc:1034] running on GCE node
I20260809 15:05:33.608196 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:33.609344 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:33.610558 22699 hybrid_clock.cc:648] HybridClock initialized: now 1786287933610495 us; error 34 us; skew 500 ppm
I20260809 15:05:33.610848 22699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:33 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:33.616636 22699 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:05:33.617931 22699 webserver.cc:460] Webserver started at http://127.27.186.190:43417/ using document root <none> and password file <none>
I20260809 15:05:33.618551 22699 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:33.618657 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:33.623487 22699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 15:05:33.626204 22721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:33.627720 22699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:05:33.627893 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "d590dd100c874a2991e84c6498fd28d1"
format_stamp: "Formatted at 2026-08-09 15:04:47 on dist-test-slave-q9h9"
server_key: "0cabe154e3f151506c16e192382e9b82"
server_key_iv: "fe1cd6496f5069f36085bf38af9e82ec"
server_key_version: "encryptionkey@0"
I20260809 15:05:33.629019 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:33.651679 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:33.655066 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:33.655301 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:33.671818 22699 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:33419
I20260809 15:05:33.671857 22807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:33419 every 8 connection(s)
I20260809 15:05:33.673120 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:05:33.674254 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 22699
I20260809 15:05:33.675341 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:43033
--local_ip_for_outbound_sockets=127.27.186.129
--tserver_master_addrs=127.27.186.190:33419
--webserver_port=36095
--webserver_interface=127.27.186.129
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:05:33.681358 22814 sys_catalog.cc:263] Verifying existing consensus state
I20260809 15:05:33.685446 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap starting.
I20260809 15:05:33.704710 22814 log.cc:824] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:33.720081 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:05:33.720552 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Bootstrap complete.
I20260809 15:05:33.728906 22814 raft_consensus.cc:348] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:33.729697 22814 raft_consensus.cc:729] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d590dd100c874a2991e84c6498fd28d1, State: Initialized, Role: FOLLOWER
I20260809 15:05:33.730293 22814 consensus_queue.cc:260] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:33.730496 22814 raft_consensus.cc:388] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:05:33.730600 22814 raft_consensus.cc:482] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:05:33.730755 22814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:05:33.734486 22814 raft_consensus.cc:504] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:33.734939 22814 leader_election.cc:302] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d590dd100c874a2991e84c6498fd28d1; no voters:
I20260809 15:05:33.735548 22814 leader_election.cc:288] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 15:05:33.735675 22824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:05:33.736737 22824 raft_consensus.cc:686] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [term 3 LEADER]: Becoming Leader. State: Replica: d590dd100c874a2991e84c6498fd28d1, State: Running, Role: LEADER
I20260809 15:05:33.737238 22824 consensus_queue.cc:237] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } }
I20260809 15:05:33.737479 22814 sys_catalog.cc:564] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:05:33.739617 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d590dd100c874a2991e84c6498fd28d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } } }
I20260809 15:05:33.739959 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:33.741498 22830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:05:33.741348 22827 sys_catalog.cc:455] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d590dd100c874a2991e84c6498fd28d1. Latest consensus state: current_term: 3 leader_uuid: "d590dd100c874a2991e84c6498fd28d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d590dd100c874a2991e84c6498fd28d1" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 33419 } } }
I20260809 15:05:33.741657 22827 sys_catalog.cc:458] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:33.749917 22830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:05:33.751344 22830 catalog_manager.cc:1259] Loaded cluster ID: a9b92d9ecabb4937b23c97ee2029888a
I20260809 15:05:33.751484 22830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:05:33.755373 22830 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:05:33.757398 22830 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Loaded TSK: 0
I20260809 15:05:33.757539 22830 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d590dd100c874a2991e84c6498fd28d1: Loaded TSK: 1
F20260809 15:05:33.758605 22830 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa9bcd229ad google::LogMessage::Fail() at ??:0
@ 0x7fa9bcd2588a google::LogMessage::SendToLog() at ??:0
@ 0x7fa9bcd224de google::LogMessage::Flush() at ??:0
@ 0x7fa9bcd23a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9c0a867d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa9c0a7573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa9d27511c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa9d2750783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa9d27dc2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa9d27dc0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa9d28265e9 std::function<>::operator()() at ??:0
@ 0x7fa9d2754cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa9d2752c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa9d27da610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa9d27da439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa9bf96e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa9bf97191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa9bf971799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa9bf94ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa9c2c266db start_thread at ??:0
@ 0x7fa9bbbbc61f clone at ??:0
W20260809 15:05:34.036248 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:34.036502 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:34.036625 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:34.043707 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:34.043815 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:34.043960 22816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:05:34.053067 22816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:43033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=36095
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:34.054174 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:34.055395 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:34.067375 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:34.067332 22892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:34.067411 22894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:34.068286 22816 server_base.cc:1034] running on GCE node
I20260809 15:05:34.068842 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:34.069661 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:34.070888 22816 hybrid_clock.cc:648] HybridClock initialized: now 1786287934070827 us; error 45 us; skew 500 ppm
I20260809 15:05:34.071189 22816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:34 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:34.074005 22816 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:05:34.075371 22816 webserver.cc:460] Webserver started at http://127.27.186.129:36095/ using document root <none> and password file <none>
I20260809 15:05:34.075976 22816 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:34.076100 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:34.081053 22816 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 15:05:34.083904 22904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:34.085129 22816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 15:05:34.085274 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "98763aaeb6a74c2684384b8f96711499"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "ee06d532ab543bcbe5b0ae23c50c2231"
server_key_iv: "15c8668738837451346b745e19b85b54"
server_key_version: "encryptionkey@0"
I20260809 15:05:34.086210 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:34.101047 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:34.106166 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:34.106441 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:34.107751 22816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:34.109915 22914 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 15:05:34.117928 22816 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 15:05:34.118074 22816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260809 15:05:34.118199 22816 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 15:05:34.122105 22816 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 15:05:34.122184 22816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.003s
I20260809 15:05:34.123432 22914 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Bootstrap starting.
I20260809 15:05:34.133306 22914 log.cc:824] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:34.149159 22914 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:05:34.149972 22914 tablet_bootstrap.cc:492] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Bootstrap complete.
I20260809 15:05:34.150569 22914 ts_tablet_manager.cc:1397] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260809 15:05:34.161211 22914 raft_consensus.cc:348] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:34.162737 22914 raft_consensus.cc:729] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98763aaeb6a74c2684384b8f96711499, State: Initialized, Role: FOLLOWER
I20260809 15:05:34.163698 22914 consensus_queue.cc:260] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:34.165448 22914 ts_tablet_manager.cc:1428] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260809 15:05:34.184051 22816 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:43033
I20260809 15:05:34.184109 23099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:43033 every 8 connection(s)
I20260809 15:05:34.185233 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:05:34.186375 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 22816
I20260809 15:05:34.187201 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:38873
--local_ip_for_outbound_sockets=127.27.186.130
--tserver_master_addrs=127.27.186.190:33419
--webserver_port=42989
--webserver_interface=127.27.186.130
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:05:34.190287 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:34.188038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:37263)
0809 15:05:34.188911 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:34.189182 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":329,"threads_started":1}
W20260809 15:05:34.190805 23100 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:33419 (0 consecutive failures): Network error: Failed to ping master at 127.27.186.190:33419: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
I20260809 15:05:34.191591 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:34.191251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:40531)
0809 15:05:34.191402 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:34.191480 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 15:05:34.192438 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:34.192112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:55667)
0809 15:05:34.192245 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:34.192325 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 15:05:34.192669 23100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 15:05:34.505117 23103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:34.505339 23103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:34.505466 23103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:34.512418 23103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:34.512514 23103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:34.512679 23103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:05:34.521723 23103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:38873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=42989
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:34.522766 23103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:34.523959 23103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:34.535768 23111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:34.535816 23112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:34.535849 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:34.536715 23103 server_base.cc:1034] running on GCE node
I20260809 15:05:34.537294 23103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:34.538136 23103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:34.539335 23103 hybrid_clock.cc:648] HybridClock initialized: now 1786287934539287 us; error 43 us; skew 500 ppm
I20260809 15:05:34.539613 23103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:34 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:34.542526 23103 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:05:34.543771 23103 webserver.cc:460] Webserver started at http://127.27.186.130:42989/ using document root <none> and password file <none>
I20260809 15:05:34.544346 23103 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:34.544445 23103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:34.549157 23103 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:05:34.551640 23132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:34.552809 23103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:05:34.552953 23103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "5f491a9798f74674a07d0b3c45aca433"
format_stamp: "Formatted at 2026-08-09 15:04:48 on dist-test-slave-q9h9"
server_key: "41aa2818fb315b307aa4892c1212a2b7"
server_key_iv: "ae9e1ea6fcd85986900ffc151bbcd532"
server_key_version: "encryptionkey@0"
I20260809 15:05:34.553819 23103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:34.569993 23103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:34.573701 23103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:34.573923 23103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:34.574869 23103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:34.576053 23103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:34.576140 23103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:34.576259 23103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:34.576333 23103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:34.605528 23103 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:38873
I20260809 15:05:34.605584 23316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:38873 every 8 connection(s)
I20260809 15:05:34.606837 23103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:05:34.611048 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:34.609386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:59977)
0809 15:05:34.610078 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:34.610289 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":258,"threads_started":1}
W20260809 15:05:34.611680 23317 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:33419 (0 consecutive failures): Network error: Failed to ping master at 127.27.186.190:33419: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
I20260809 15:05:34.612404 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:34.612090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:60219)
0809 15:05:34.612226 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:34.612295 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 15:05:34.613183 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:34.612870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:50247)
0809 15:05:34.613018 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:34.613084 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 15:05:34.613376 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 23103
W20260809 15:05:34.613401 23317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 15:05:34.613966 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:34517
--local_ip_for_outbound_sockets=127.27.186.131
--tserver_master_addrs=127.27.186.190:33419
--webserver_port=41017
--webserver_interface=127.27.186.131
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:34.932031 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:34.932278 23323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:34.932407 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:34.939254 23323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:34.939343 23323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:34.939486 23323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:05:34.948410 23323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:34517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=41017
--tserver_master_addrs=127.27.186.190:33419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:34.949474 23323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:34.950637 23323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:34.962715 23330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:34.962754 23334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:34.962786 23331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:34.963590 23323 server_base.cc:1034] running on GCE node
I20260809 15:05:34.964128 23323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:34.964998 23323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:34.966208 23323 hybrid_clock.cc:648] HybridClock initialized: now 1786287934966143 us; error 40 us; skew 500 ppm
I20260809 15:05:34.966492 23323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:34 dist-test-slave-q9h9 krb5kdc[20934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:34.969434 23323 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:05:34.970753 23323 webserver.cc:460] Webserver started at http://127.27.186.131:41017/ using document root <none> and password file <none>
I20260809 15:05:34.971365 23323 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:34.971467 23323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:34.976418 23323 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 15:05:34.978987 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:34.980176 23323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:05:34.980320 23323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2"
format_stamp: "Formatted at 2026-08-09 15:04:49 on dist-test-slave-q9h9"
server_key: "eafa08188c8b238dfd347631c53d75f1"
server_key_iv: "27d9541cd09478a72cbb1b3cad1860a5"
server_key_version: "encryptionkey@0"
I20260809 15:05:34.981215 23323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:34.994927 23323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:34.998754 23323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:34.998967 23323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:34.999804 23323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:35.000986 23323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:35.001072 23323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:35.001147 23323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:35.001226 23323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:35.030644 23323 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:34517
I20260809 15:05:35.030720 23527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:34517 every 8 connection(s)
I20260809 15:05:35.031924 23323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:05:35.036175 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.034514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:34989)
0809 15:05:35.035187 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.035402 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":260,"threads_started":1}
W20260809 15:05:35.036716 23528 heartbeater.cc:644] Failed to heartbeat to 127.27.186.190:33419 (0 consecutive failures): Network error: Failed to ping master at 127.27.186.190:33419: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
I20260809 15:05:35.037413 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.037135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:41279)
0809 15:05:35.037247 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.037310 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 15:05:35.038132 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.037831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:45247)
0809 15:05:35.037931 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.038000 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 15:05:35.038340 23528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 15:05:35.039011 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 23323
I20260809 15:05:35.193792 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.193108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:35189)
0809 15:05:35.193528 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.193603 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":190,"threads_started":1}
I20260809 15:05:35.396250 23534 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:35.396523 23534 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:35.398124 23534 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
Aug 09 15:05:35 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
Aug 09 15:05:35 dist-test-slave-q9h9 krb5kdc[20934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:05:35.417917 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.398491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34517 (local address 127.27.186.129:42243)
0809 15:05:35.399131 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.399174 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:35.399786 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:35.402207 (+ 2421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:35.402218 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:35.403032 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:35.403774 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:35.403792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.405290 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.405294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:35.406745 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:35.406757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.407653 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.407660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:35.407919 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:35.408644 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:35.408660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:35.410864 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:35.414533 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:35.414539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:35.414549 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:35.414975 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:35.415376 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:35.415380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:35.415384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:35.415581 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:35.416451 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:35.416456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:35.416737 (+ 281us) client_negotiation.cc:769] Sending connection context
0809 15:05:35.417046 (+ 309us) client_negotiation.cc:240] Negotiation successful
0809 15:05:35.417488 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":189,"spinlock_wait_cycles":54784}
I20260809 15:05:35.418038 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.398532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38873 (local address 127.27.186.129:49125)
0809 15:05:35.399431 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.399450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:35.399794 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:35.402653 (+ 2859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:35.402658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:35.403135 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:35.403773 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:35.403793 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.405305 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.405309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:35.406737 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:35.406750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.407704 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.407712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:35.407917 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:35.408591 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:35.408613 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:35.411248 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:35.414975 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:35.414980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:35.414984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:35.415345 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:35.415732 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:35.415735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:35.415738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:35.415852 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:35.416692 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:35.416696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:35.416813 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 15:05:35.417046 (+ 233us) client_negotiation.cc:240] Negotiation successful
0809 15:05:35.417630 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":284,"spinlock_wait_cycles":26496,"thread_start_us":276,"threads_started":1}
I20260809 15:05:35.418087 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.398635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:42243 (local address 127.27.186.131:34517)
0809 15:05:35.399900 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:35.399914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:35.399969 (+ 55us) server_negotiation.cc:407] Connection header received
0809 15:05:35.401455 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:35.401464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:35.401783 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:35.402078 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:35.403956 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.405110 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.406873 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.407454 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.407627 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:35.411134 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:35.411142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:35.411154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:35.411187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:35.414404 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:35.415103 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:35.415110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:35.415115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:35.415267 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:35.415698 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:35.415703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:35.415706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:35.416323 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:35.416510 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:35.417278 (+ 768us) server_negotiation.cc:299] Negotiation successful
0809 15:05:35.417658 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":921,"threads_started":1}
I20260809 15:05:35.418593 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.398780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:49125 (local address 127.27.186.130:38873)
0809 15:05:35.399368 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:35.399383 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:35.399776 (+ 393us) server_negotiation.cc:407] Connection header received
0809 15:05:35.401406 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:35.401414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:35.401752 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:35.402060 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:35.403953 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.405136 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.406866 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:35.407461 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:35.407621 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:35.411483 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:35.411490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:35.411501 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:35.411538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:35.414847 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:35.415472 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:35.415478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:35.415483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:35.415625 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:35.415969 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:35.415974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:35.415977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:35.416581 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:35.416711 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:35.417292 (+ 581us) server_negotiation.cc:299] Negotiation successful
0809 15:05:35.417667 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":271,"threads_started":1}
W20260809 15:05:35.420948 22906 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
W20260809 15:05:35.421200 22907 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:35.421317 22907 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98763aaeb6a74c2684384b8f96711499; no voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2
I20260809 15:05:35.421650 23534 raft_consensus.cc:2726] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:05:35.614611 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:35.613830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:41277)
0809 15:05:35.614253 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:35.614391 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":204,"threads_started":1}
I20260809 15:05:36.039448 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:36.038747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:36329)
0809 15:05:36.039145 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:36.039262 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":193,"threads_started":1}
I20260809 15:05:36.195219 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:36.194586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:57845)
0809 15:05:36.194993 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:36.195070 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":205,"threads_started":1}
I20260809 15:05:36.615965 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:36.615366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:34405)
0809 15:05:36.615751 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:36.615826 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":202,"threads_started":1}
I20260809 15:05:37.040930 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:37.040235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:58633)
0809 15:05:37.040665 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:37.040753 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260809 15:05:37.196609 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:37.195928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:37817)
0809 15:05:37.196368 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:37.196444 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260809 15:05:37.617434 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:37.616743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:33879)
0809 15:05:37.617200 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:37.617279 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260809 15:05:37.665515 23549 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:37.665794 23549 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:37.666608 23549 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
W20260809 15:05:37.668412 22907 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
W20260809 15:05:37.668584 22906 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:37.668772 22906 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98763aaeb6a74c2684384b8f96711499; no voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2
I20260809 15:05:37.669106 23549 raft_consensus.cc:2726] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:05:38.042410 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:38.041673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:37115)
0809 15:05:38.042163 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:38.042253 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":229,"threads_started":1}
I20260809 15:05:38.198112 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:38.197436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:34967)
0809 15:05:38.197854 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:38.197931 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260809 15:05:38.618943 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:38.618227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:34981)
0809 15:05:38.618646 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:38.618747 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":200,"threads_started":1}
I20260809 15:05:39.043874 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:39.043177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:33949)
0809 15:05:39.043607 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:39.043687 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260809 15:05:39.199831 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:39.198997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:54845)
0809 15:05:39.199534 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:39.199640 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":261,"threads_started":1}
I20260809 15:05:39.339157 23560 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:39.339368 23560 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:39.340171 23560 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
W20260809 15:05:39.341002 22907 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
W20260809 15:05:39.341159 22906 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:39.341269 22906 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98763aaeb6a74c2684384b8f96711499; no voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2
I20260809 15:05:39.341499 23560 raft_consensus.cc:2726] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:05:39.620388 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:39.619730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:56983)
0809 15:05:39.620145 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:39.620223 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":225,"threads_started":1}
I20260809 15:05:40.045333 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:40.044687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:56195)
0809 15:05:40.045112 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:40.045195 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":225,"threads_started":1}
I20260809 15:05:40.201285 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:40.200625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:48955)
0809 15:05:40.201032 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:40.201141 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260809 15:05:40.621784 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:40.621135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:54725)
0809 15:05:40.621537 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:40.621621 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":209,"threads_started":1}
I20260809 15:05:41.046849 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:41.046128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:38797)
0809 15:05:41.046579 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:41.046662 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260809 15:05:41.202723 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:41.202011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:54967)
0809 15:05:41.202458 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:41.202545 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260809 15:05:41.623291 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:41.622559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:53147)
0809 15:05:41.623030 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:41.623107 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":214,"threads_started":1}
I20260809 15:05:42.048393 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:42.047598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:59613)
0809 15:05:42.048119 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:42.048196 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":232,"threads_started":1}
I20260809 15:05:42.204154 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:42.203498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:50839)
0809 15:05:42.203922 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:42.204011 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":205,"threads_started":1}
I20260809 15:05:42.624677 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:42.624017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:50377)
0809 15:05:42.624411 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:42.624487 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":206,"threads_started":1}
I20260809 15:05:43.049906 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:43.049190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:58387)
0809 15:05:43.049646 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:43.049729 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":253,"threads_started":1}
I20260809 15:05:43.205587 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:43.204927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:41235)
0809 15:05:43.205349 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:43.205427 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260809 15:05:43.401783 23581 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:43.402002 23581 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:43.402665 23581 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
W20260809 15:05:43.403589 22906 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
W20260809 15:05:43.403800 22907 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:43.403941 22907 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98763aaeb6a74c2684384b8f96711499; no voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2
I20260809 15:05:43.404194 23581 raft_consensus.cc:2726] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:05:43.626307 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:43.625443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:36885)
0809 15:05:43.625993 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:43.626118 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":250,"threads_started":1}
I20260809 15:05:44.051333 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:44.050645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:51915)
0809 15:05:44.051082 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:44.051182 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260809 15:05:44.207110 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:44.206441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:41465)
0809 15:05:44.206881 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:44.206960 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260809 15:05:44.627728 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:44.627101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:42601)
0809 15:05:44.627482 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:44.627563 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":202,"threads_started":1}
I20260809 15:05:45.052759 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:45.052098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:39701)
0809 15:05:45.052496 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:45.052570 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":202,"threads_started":1}
I20260809 15:05:45.208753 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:45.207941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:59795)
0809 15:05:45.208471 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:45.208565 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":279,"threads_started":1}
I20260809 15:05:45.629382 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:45.628549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:48779)
0809 15:05:45.629086 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:45.629191 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":263,"threads_started":1}
I20260809 15:05:46.054330 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:46.053635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:49583)
0809 15:05:46.054090 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:46.054166 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260809 15:05:46.210240 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:46.209516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:46039)
0809 15:05:46.209942 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:46.210077 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260809 15:05:46.630925 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:46.630233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:42473)
0809 15:05:46.630685 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:46.630764 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260809 15:05:46.748332 23602 raft_consensus.cc:482] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:46.748507 23602 raft_consensus.cc:504] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa887bb4a7b4f918534ab9d3eb0b7c2" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34517 } } peers { permanent_uuid: "98763aaeb6a74c2684384b8f96711499" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43033 } } peers { permanent_uuid: "5f491a9798f74674a07d0b3c45aca433" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38873 } }
I20260809 15:05:46.749146 23602 leader_election.cc:288] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517), 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873)
W20260809 15:05:46.750090 22907 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa887bb4a7b4f918534ab9d3eb0b7c2 (127.27.186.131:34517): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
W20260809 15:05:46.750285 22906 leader_election.cc:341] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f491a9798f74674a07d0b3c45aca433 (127.27.186.130:38873): Not found: Tablet not found: 0de4826bcccc4c65885544abecc90b6c
I20260809 15:05:46.750382 22906 leader_election.cc:302] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98763aaeb6a74c2684384b8f96711499; no voters: 5f491a9798f74674a07d0b3c45aca433, 6fa887bb4a7b4f918534ab9d3eb0b7c2
I20260809 15:05:46.750614 23602 raft_consensus.cc:2726] T 0de4826bcccc4c65885544abecc90b6c P 98763aaeb6a74c2684384b8f96711499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:05:47.055698 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:47.055059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:36353)
0809 15:05:47.055456 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:47.055539 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260809 15:05:47.211704 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:47.210962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:57857)
0809 15:05:47.211393 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:47.211484 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":211,"threads_started":1}
I20260809 15:05:47.632678 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:47.631807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:42653)
0809 15:05:47.632383 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:47.632500 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
I20260809 15:05:48.057248 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:48.056515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:41479)
0809 15:05:48.056952 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:48.057089 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":219,"threads_started":1}
I20260809 15:05:48.213174 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:48.212480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:46399)
0809 15:05:48.212907 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:48.212997 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260809 15:05:48.634162 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:48.633444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:60679)
0809 15:05:48.633869 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:48.633946 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260809 15:05:49.058763 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:49.058072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:35311)
0809 15:05:49.058507 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:49.058583 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260809 15:05:49.214905 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:49.214209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.129:33033)
0809 15:05:49.214634 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:49.214724 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":203,"threads_started":1}
I20260809 15:05:49.635804 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:49.635128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.130:51669)
0809 15:05:49.635525 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:49.635641 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260809 15:05:50.049489 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 22816
I20260809 15:05:50.060451 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:50.059702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:33419 (local address 127.27.186.131:47533)
0809 15:05:50.060187 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:50.060293 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.190:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":240,"threads_started":1}
I20260809 15:05:50.069065 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:50.050475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.27.186.129:43033)
0809 15:05:50.051060 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:50.051071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:50.051107 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:05:50.051341 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:50.051351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:50.051650 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:50.051843 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:50.053290 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:50.054318 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:50.055535 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:50.056098 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:50.056190 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:50.063647 (+ 7457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:50.063657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:50.063669 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:50.063709 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:50.066449 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:50.067158 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:50.067165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:50.067167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:50.067312 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:50.067645 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:50.067648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:50.067650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:50.068006 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:50.068244 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:50.068548 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 15:05:50.068763 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260809 15:05:50.070549 23094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:50.158808 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 22816
I20260809 15:05:50.197144 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 23103
I20260809 15:05:50.209411 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:50.197871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.27.186.130:38873)
0809 15:05:50.198334 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:50.198339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:50.198355 (+ 16us) server_negotiation.cc:407] Connection header received
0809 15:05:50.198410 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:50.198413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:50.198474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:50.198567 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:50.199852 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:50.200606 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:50.201761 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:50.202245 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:50.202334 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:50.204687 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:50.204691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:50.204694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:50.204725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:50.207238 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:50.207837 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:50.207840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:50.207842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:50.207974 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:50.208257 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:50.208260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:50.208262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:50.208510 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:50.208715 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:50.208947 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 15:05:50.209111 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260809 15:05:50.210150 23310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:50.293877 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 23103
I20260809 15:05:50.321233 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 23323
I20260809 15:05:50.334543 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:50.322156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.27.186.131:34517)
0809 15:05:50.322710 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:50.322715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:50.322731 (+ 16us) server_negotiation.cc:407] Connection header received
0809 15:05:50.322800 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:50.322803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:50.322868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:50.322950 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:50.324298 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:50.325083 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:50.326306 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:50.326704 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:50.326800 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:50.329349 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:50.329354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:50.329357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:50.329390 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:50.331966 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:50.332606 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:50.332612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:50.332614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:50.332752 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:50.333108 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:50.333112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:50.333114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:50.333385 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:50.333652 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:50.333918 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 15:05:50.334199 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":291,"threads_started":1}
I20260809 15:05:50.335237 23523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:50.417052 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 23323
2026-08-09T15:05:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64594 ms)
[----------] 30 tests from SecurityITest (290312 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:05:50 dist-test-slave-q9h9 krb5kdc[23636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:05:50 dist-test-slave-q9h9 krb5kdc[23636](info): set up 2 sockets
Aug 09 15:05:50 dist-test-slave-q9h9 krb5kdc[23636](info): commencing operation
krb5kdc: starting...
W20260809 15:05:51.641526 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:05:51 dist-test-slave-q9h9 krb5kdc[23636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:05:51.867410 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:45413
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:45413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:52.184075 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:52.184325 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:52.184417 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:52.191344 23653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:05:52.191433 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:52.191484 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:52.191537 23653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:05:52.191582 23653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:05:52.200598 23653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:45413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:45413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:52.201625 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:52.202802 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:52.213522 23668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:52.213562 23671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:52.213588 23666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:52.214108 23653 server_base.cc:1034] running on GCE node
I20260809 15:05:52.214876 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:52.216065 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:52.217311 23653 hybrid_clock.cc:648] HybridClock initialized: now 1786287952217239 us; error 44 us; skew 500 ppm
I20260809 15:05:52.217633 23653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:52 dist-test-slave-q9h9 krb5kdc[23636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:52.220456 23653 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:05:52.221791 23653 webserver.cc:460] Webserver started at http://127.27.186.190:35543/ using document root <none> and password file <none>
I20260809 15:05:52.222443 23653 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:52.222561 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:52.222834 23653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:52.225538 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "e6399c74ab8847eaaa9b30f37f7c89fe"
format_stamp: "Formatted at 2026-08-09 15:05:52 on dist-test-slave-q9h9"
server_key: "d81531bac545ff47a09ee97d19aa60ff"
server_key_iv: "673ae98ab97ea7e6a55f1f83be18809f"
server_key_version: "encryptionkey@0"
I20260809 15:05:52.226281 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "e6399c74ab8847eaaa9b30f37f7c89fe"
format_stamp: "Formatted at 2026-08-09 15:05:52 on dist-test-slave-q9h9"
server_key: "d81531bac545ff47a09ee97d19aa60ff"
server_key_iv: "673ae98ab97ea7e6a55f1f83be18809f"
server_key_version: "encryptionkey@0"
I20260809 15:05:52.230989 23653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:05:52.234175 23682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:52.235337 23653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:05:52.235487 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "e6399c74ab8847eaaa9b30f37f7c89fe"
format_stamp: "Formatted at 2026-08-09 15:05:52 on dist-test-slave-q9h9"
server_key: "d81531bac545ff47a09ee97d19aa60ff"
server_key_iv: "673ae98ab97ea7e6a55f1f83be18809f"
server_key_version: "encryptionkey@0"
I20260809 15:05:52.235651 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:52.248988 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:52.252738 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:52.252983 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:52.271929 23653 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:45413
I20260809 15:05:52.271948 23771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:45413 every 8 connection(s)
I20260809 15:05:52.273178 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:05:52.275533 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 23653
I20260809 15:05:52.275759 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:05:52.276587 28394 external_mini_cluster.cc:1428] Setting key f23f1b90ef6fd56d8ab4c35733804ad5
I20260809 15:05:52.277311 23772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:52.290773 23772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe: Bootstrap starting.
Aug 09 15:05:52 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:52.293829 23772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:52.294899 23772 log.cc:824] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:52.297420 23772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe: No bootstrap required, opened a new log
I20260809 15:05:52.299889 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:52.278677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.27.186.190:45413)
0809 15:05:52.279391 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:52.279401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:52.279439 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:05:52.281692 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:52.281711 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:52.282100 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:52.282430 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:52.284258 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:52.285465 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:52.287570 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:52.288245 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:52.288379 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:52.293157 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:52.293164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:52.293177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:52.293212 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:52.296104 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:52.296758 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:52.296764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:52.296770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:52.296934 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:52.297297 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:52.297300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:52.297302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:52.297834 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:52.298134 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:52.298552 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 15:05:52.298929 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260809 15:05:52.305492 23772 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45413 } }
I20260809 15:05:52.305917 23772 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:52.306041 23772 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6399c74ab8847eaaa9b30f37f7c89fe, State: Initialized, Role: FOLLOWER
I20260809 15:05:52.306524 23772 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45413 } }
I20260809 15:05:52.306717 23772 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:05:52.306809 23772 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:05:52.306929 23772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:52.309120 23772 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45413 } }
I20260809 15:05:52.309547 23772 leader_election.cc:302] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6399c74ab8847eaaa9b30f37f7c89fe; no voters:
I20260809 15:05:52.310077 23772 leader_election.cc:288] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:05:52.310149 23780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:05:52.310798 23780 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [term 1 LEADER]: Becoming Leader. State: Replica: e6399c74ab8847eaaa9b30f37f7c89fe, State: Running, Role: LEADER
I20260809 15:05:52.311339 23780 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45413 } }
I20260809 15:05:52.311935 23772 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:05:52.314049 23786 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6399c74ab8847eaaa9b30f37f7c89fe. Latest consensus state: current_term: 1 leader_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45413 } } }
I20260809 15:05:52.314112 23784 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6399c74ab8847eaaa9b30f37f7c89fe" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45413 } } }
I20260809 15:05:52.314528 23784 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:52.315167 23786 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:52.315943 23795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:05:52.320050 23795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:05:52.327581 23795 catalog_manager.cc:1347] Generated new cluster ID: 7e4dffe5c5d34e5c93ac43e56d36ba65
I20260809 15:05:52.327687 23795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:05:52.334683 23795 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:05:52.336321 23795 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:05:52.350462 23795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe: Generated new TSK 0
I20260809 15:05:52.351344 23795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:05:52.414913 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45413
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:52.734417 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:52.734674 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:52.734809 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:52.741847 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:52.741952 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:52.742116 23819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:05:52.751174 23819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:52.752264 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:52.753444 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:52.765853 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:52.765904 23828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:52.766171 23826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:52.766841 23819 server_base.cc:1034] running on GCE node
I20260809 15:05:52.767345 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:52.768133 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:52.769335 23819 hybrid_clock.cc:648] HybridClock initialized: now 1786287952769292 us; error 38 us; skew 500 ppm
I20260809 15:05:52.769594 23819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:52 dist-test-slave-q9h9 krb5kdc[23636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:52.772456 23819 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:05:52.773784 23819 webserver.cc:460] Webserver started at http://127.27.186.129:46197/ using document root <none> and password file <none>
I20260809 15:05:52.774433 23819 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:52.774552 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:52.774864 23819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:52.777571 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe"
format_stamp: "Formatted at 2026-08-09 15:05:52 on dist-test-slave-q9h9"
server_key: "2906e2b705477a4f5e4fa5887ab3dc43"
server_key_iv: "af1b9c2769a663f6e80a28edfe3b0721"
server_key_version: "encryptionkey@0"
I20260809 15:05:52.778326 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe"
format_stamp: "Formatted at 2026-08-09 15:05:52 on dist-test-slave-q9h9"
server_key: "2906e2b705477a4f5e4fa5887ab3dc43"
server_key_iv: "af1b9c2769a663f6e80a28edfe3b0721"
server_key_version: "encryptionkey@0"
I20260809 15:05:52.782990 23819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:05:52.786134 23846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:52.787243 23819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 15:05:52.787395 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe"
format_stamp: "Formatted at 2026-08-09 15:05:52 on dist-test-slave-q9h9"
server_key: "2906e2b705477a4f5e4fa5887ab3dc43"
server_key_iv: "af1b9c2769a663f6e80a28edfe3b0721"
server_key_version: "encryptionkey@0"
I20260809 15:05:52.787572 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:52.815251 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:52.818881 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:52.819130 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:52.819975 23819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:52.821214 23819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:52.821302 23819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:52.821416 23819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:52.821480 23819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:52.851076 23819 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:40915
I20260809 15:05:52.851147 24042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:40915 every 8 connection(s)
I20260809 15:05:52.852221 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:05:52.857030 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 23819
I20260809 15:05:52.857208 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:05:52.857831 28394 external_mini_cluster.cc:1428] Setting key 032cc89d2f6d506574658fa25099f669
Aug 09 15:05:52 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:52.871366 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:52.854429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:40681 (local address 127.27.186.190:45413)
0809 15:05:52.854848 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:52.854853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:52.855731 (+ 878us) server_negotiation.cc:407] Connection header received
0809 15:05:52.857511 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:52.857517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:52.857596 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:52.857691 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:52.859262 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:52.860072 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:52.862201 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:52.862585 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:52.862682 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:52.866158 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:52.866163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:52.866166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:52.866197 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:52.868534 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:52.869203 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:52.869207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:52.869217 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:52.869338 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:52.869743 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:52.869746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:52.869748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:52.869991 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:52.870163 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:52.870785 (+ 622us) server_negotiation.cc:299] Negotiation successful
0809 15:05:52.871001 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":195,"threads_started":1}
I20260809 15:05:52.872161 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:52.854945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:45413 (local address 127.27.186.129:40681)
0809 15:05:52.855578 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:52.855614 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:52.857244 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:52.857963 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:52.857973 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:52.858464 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:52.859098 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:52.859123 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:52.860232 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:52.860238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:52.862075 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:52.862089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:52.862800 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:52.862807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:52.863069 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:52.863668 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:52.863695 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:52.865958 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:52.868673 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:52.868682 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:52.868696 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:52.869082 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:52.869440 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:52.869445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:52.869449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:52.869629 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:52.870138 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:52.870145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:52.870433 (+ 288us) client_negotiation.cc:769] Sending connection context
0809 15:05:52.870642 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 15:05:52.871103 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":255,"threads_started":1}
I20260809 15:05:52.873600 24044 heartbeater.cc:347] Connected to a master server at 127.27.186.190:45413
I20260809 15:05:52.873853 24044 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:52.874516 24044 heartbeater.cc:511] Master 127.27.186.190:45413 requested a full tablet report, sending...
I20260809 15:05:52.876566 23709 ts_manager.cc:194] Registered new tserver with Master: b8c6db248ca34ac0b5ff8cc7e1373bfe (127.27.186.129:40915)
I20260809 15:05:52.878482 23709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:40681
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:05:52.918594 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45413
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:53.238545 24054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:53.238780 24054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:53.238883 24054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:53.245782 24054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:53.245873 24054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:53.245985 24054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:05:53.255236 24054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:53.256345 24054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:53.257599 24054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:53.270056 24064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:53.270102 24070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:53.270113 24065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:53.270854 24054 server_base.cc:1034] running on GCE node
I20260809 15:05:53.271466 24054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:53.272496 24054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:53.273732 24054 hybrid_clock.cc:648] HybridClock initialized: now 1786287953273666 us; error 60 us; skew 500 ppm
I20260809 15:05:53.274080 24054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:53 dist-test-slave-q9h9 krb5kdc[23636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:53.277789 24054 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:05:53.279371 24054 webserver.cc:460] Webserver started at http://127.27.186.130:39561/ using document root <none> and password file <none>
I20260809 15:05:53.280149 24054 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:53.280270 24054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:53.280578 24054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:53.284106 24054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "e201b00dd3424bda9f6428753f774722"
format_stamp: "Formatted at 2026-08-09 15:05:53 on dist-test-slave-q9h9"
server_key: "f568d021cabf8a8a8e0d5d646b2ad47e"
server_key_iv: "2b6ceb22f0c2908f3254812f0b1099e8"
server_key_version: "encryptionkey@0"
I20260809 15:05:53.285017 24054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "e201b00dd3424bda9f6428753f774722"
format_stamp: "Formatted at 2026-08-09 15:05:53 on dist-test-slave-q9h9"
server_key: "f568d021cabf8a8a8e0d5d646b2ad47e"
server_key_iv: "2b6ceb22f0c2908f3254812f0b1099e8"
server_key_version: "encryptionkey@0"
I20260809 15:05:53.291115 24054 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260809 15:05:53.294230 24080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:53.295387 24054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 15:05:53.295524 24054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "e201b00dd3424bda9f6428753f774722"
format_stamp: "Formatted at 2026-08-09 15:05:53 on dist-test-slave-q9h9"
server_key: "f568d021cabf8a8a8e0d5d646b2ad47e"
server_key_iv: "2b6ceb22f0c2908f3254812f0b1099e8"
server_key_version: "encryptionkey@0"
I20260809 15:05:53.295696 24054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:53.321911 24054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:53.325448 24054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:53.325701 24054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:53.326643 24054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:53.327860 24054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:53.327934 24054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:53.328043 24054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:53.328091 24054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:53.358215 24054 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:35019
I20260809 15:05:53.358258 24262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:35019 every 8 connection(s)
I20260809 15:05:53.359325 24054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:05:53.369349 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 24054
I20260809 15:05:53.369534 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:05:53.370091 28394 external_mini_cluster.cc:1428] Setting key df42fa0be095a0a0a427774e4100fe54
Aug 09 15:05:53 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:53.377517 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.361492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:60813 (local address 127.27.186.190:45413)
0809 15:05:53.361680 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:53.361687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:53.362777 (+ 1090us) server_negotiation.cc:407] Connection header received
0809 15:05:53.364481 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:53.364485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:53.364551 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:53.364633 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:53.365926 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.366669 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.368052 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.368461 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.368545 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.372179 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:53.372184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:53.372187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:53.372217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:53.374744 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:53.375523 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:53.375528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:53.375530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:53.375658 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:53.376059 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:53.376062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:53.376064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:53.376307 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:53.376414 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:53.377111 (+ 697us) server_negotiation.cc:299] Negotiation successful
0809 15:05:53.377289 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:05:53.378639 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.361920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:45413 (local address 127.27.186.130:60813)
0809 15:05:53.362613 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:53.362655 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:53.364258 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:53.364807 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:53.364814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:53.365264 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:53.365778 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.365795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.366782 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.366787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.367945 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.367962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.368622 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.368627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.368822 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.369394 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:53.369424 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:53.371964 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:53.374879 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:53.374887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:53.374900 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:53.375404 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:53.375754 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:53.375758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:53.375761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:53.375975 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:53.376430 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:53.376436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:53.376786 (+ 350us) client_negotiation.cc:769] Sending connection context
0809 15:05:53.377050 (+ 264us) client_negotiation.cc:240] Negotiation successful
0809 15:05:53.377480 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":265,"threads_started":1}
I20260809 15:05:53.379894 24263 heartbeater.cc:347] Connected to a master server at 127.27.186.190:45413
I20260809 15:05:53.380178 24263 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:53.380908 24263 heartbeater.cc:511] Master 127.27.186.190:45413 requested a full tablet report, sending...
I20260809 15:05:53.382277 23709 ts_manager.cc:194] Registered new tserver with Master: e201b00dd3424bda9f6428753f774722 (127.27.186.130:35019)
I20260809 15:05:53.383039 23709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:60813
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:05:53.427310 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45413
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:53.749126 24268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:53.749365 24268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:53.749457 24268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:53.756369 24268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:53.756461 24268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:53.756573 24268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:05:53.765698 24268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:53.766745 24268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:53.767949 24268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:53.780102 24273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:53.780133 24275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:53.780129 24279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:53.780805 24268 server_base.cc:1034] running on GCE node
I20260809 15:05:53.781347 24268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:53.782212 24268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:53.783449 24268 hybrid_clock.cc:648] HybridClock initialized: now 1786287953783391 us; error 41 us; skew 500 ppm
I20260809 15:05:53.783742 24268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:53 dist-test-slave-q9h9 krb5kdc[23636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:53.786649 24268 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:05:53.787962 24268 webserver.cc:460] Webserver started at http://127.27.186.131:33089/ using document root <none> and password file <none>
I20260809 15:05:53.788570 24268 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:53.788686 24268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:53.788981 24268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:53.791625 24268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "451648ae8b6b424d9306a3e9f3835fab"
format_stamp: "Formatted at 2026-08-09 15:05:53 on dist-test-slave-q9h9"
server_key: "1240d6399cf2289c699615e43650f610"
server_key_iv: "dc05b082d7a4627ca9cbc306647b310b"
server_key_version: "encryptionkey@0"
I20260809 15:05:53.792340 24268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "451648ae8b6b424d9306a3e9f3835fab"
format_stamp: "Formatted at 2026-08-09 15:05:53 on dist-test-slave-q9h9"
server_key: "1240d6399cf2289c699615e43650f610"
server_key_iv: "dc05b082d7a4627ca9cbc306647b310b"
server_key_version: "encryptionkey@0"
I20260809 15:05:53.797030 24268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260809 15:05:53.800369 24289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:53.801606 24268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 15:05:53.801748 24268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "451648ae8b6b424d9306a3e9f3835fab"
format_stamp: "Formatted at 2026-08-09 15:05:53 on dist-test-slave-q9h9"
server_key: "1240d6399cf2289c699615e43650f610"
server_key_iv: "dc05b082d7a4627ca9cbc306647b310b"
server_key_version: "encryptionkey@0"
I20260809 15:05:53.801916 24268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:53.814268 24268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:53.817932 24268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:53.818200 24268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:53.819023 24268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:53.820214 24268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:53.820286 24268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:53.820391 24268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:53.820446 24268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:53.850844 24268 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:38641
I20260809 15:05:53.850862 24466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:38641 every 8 connection(s)
I20260809 15:05:53.852317 24268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:05:53.858314 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 24268
I20260809 15:05:53.858484 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:05:53.858955 28394 external_mini_cluster.cc:1428] Setting key 386afc13b6d802b643bc3fce1c7adc3a
Aug 09 15:05:53 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:53.872669 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.855058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:60193 (local address 127.27.186.190:45413)
0809 15:05:53.855256 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:53.855263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:53.858803 (+ 3540us) server_negotiation.cc:407] Connection header received
0809 15:05:53.860543 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:53.860548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:53.860614 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:53.860720 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:53.862187 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.862923 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.864326 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.864719 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.864815 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.867871 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:53.867875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:53.867878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:53.867909 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:53.870148 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:53.870797 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:53.870801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:53.870803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:53.870932 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:53.871277 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:53.871280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:53.871282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:53.871517 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:53.871623 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:53.872199 (+ 576us) server_negotiation.cc:299] Negotiation successful
0809 15:05:53.872377 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:05:53.873607 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.855385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:45413 (local address 127.27.186.131:60193)
0809 15:05:53.858625 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:53.858683 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:53.860286 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:53.860939 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:53.860946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:53.861401 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:53.862008 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.862065 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.863038 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.863042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.864225 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.864237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.864875 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.864879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.865063 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.865660 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:53.865688 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:53.867702 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:53.870294 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:53.870301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:53.870310 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:53.870696 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:53.871009 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:53.871012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:53.871015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:53.871196 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:53.871602 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:53.871608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:53.871864 (+ 256us) client_negotiation.cc:769] Sending connection context
0809 15:05:53.872057 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 15:05:53.872490 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"thread_start_us":2761,"threads_started":1}
I20260809 15:05:53.874902 24467 heartbeater.cc:347] Connected to a master server at 127.27.186.190:45413
I20260809 15:05:53.875174 24467 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:53.875799 24467 heartbeater.cc:511] Master 127.27.186.190:45413 requested a full tablet report, sending...
I20260809 15:05:53.877066 23709 ts_manager.cc:194] Registered new tserver with Master: 451648ae8b6b424d9306a3e9f3835fab (127.27.186.131:38641)
I20260809 15:05:53.877792 23709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:60193
I20260809 15:05:53.881069 24044 heartbeater.cc:503] Master 127.27.186.190:45413 was elected leader, sending a full tablet report...
I20260809 15:05:53.885860 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:05:53.902136 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.890122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.27.186.190:45413)
0809 15:05:53.890253 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:53.890258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:53.890735 (+ 477us) server_negotiation.cc:407] Connection header received
0809 15:05:53.890880 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:53.890885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:53.890942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:53.891019 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:53.892422 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.893094 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.894410 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.894811 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.894935 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.897148 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:53.897153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:53.897156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:53.897185 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:53.899501 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:53.900265 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:53.900271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:53.900273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:53.900399 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:53.900767 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:53.900770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:53.900772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:53.901053 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:53.901291 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:53.901634 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 15:05:53.901828 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:05:53.905568 23709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:05:53.912194 23709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:05:53.944226 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.923145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:35019 (local address 127.0.0.1:47892)
0809 15:05:53.936720 (+ 13575us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:53.936736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:53.936831 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:53.937891 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:53.937899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:53.937927 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:53.938451 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.938463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.940247 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.940252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.941716 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.941729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.943399 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.943406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.943579 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.943618 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 15:05:53.943725 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 15:05:53.943886 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13455,"thread_start_us":13331,"threads_started":1}
I20260809 15:05:53.944260 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.923638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.27.186.130:35019)
0809 15:05:53.937361 (+ 13723us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:53.937369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:53.937395 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:05:53.937483 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:53.937490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:53.937703 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:53.937895 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:53.938638 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.940154 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.941864 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.943170 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.943346 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.943498 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:53.943792 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 15:05:53.943947 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13564,"thread_start_us":13413,"threads_started":1}
I20260809 15:05:53.945040 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.922961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:38641 (local address 127.0.0.1:56144)
0809 15:05:53.936134 (+ 13173us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:53.936182 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:53.936442 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:53.937476 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:53.937483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:53.937511 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:53.937931 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.937945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.939931 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.939937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.942240 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.942255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.944172 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.944178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.944467 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.944501 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:05:53.944590 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 15:05:53.944716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12967,"thread_start_us":12832,"threads_started":1}
I20260809 15:05:53.945875 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.923121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.27.186.131:38641)
0809 15:05:53.935483 (+ 12362us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:53.935492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:53.936278 (+ 786us) server_negotiation.cc:407] Connection header received
0809 15:05:53.936721 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:53.936728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:53.936960 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:53.937143 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:53.938170 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.939767 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.942394 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.943705 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.943875 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.944026 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:53.944676 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 15:05:53.945585 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12173,"thread_start_us":12379,"threads_started":1}
I20260809 15:05:53.946251 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.937757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:40915 (local address 127.0.0.1:49486)
0809 15:05:53.938692 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:53.938709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:53.938827 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:53.939585 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:53.939590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:53.939615 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:53.940016 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.940027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.941925 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.941930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.943440 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:53.943453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.945530 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.945536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:53.945691 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.945722 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:05:53.945816 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 15:05:53.945914 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":23,"thread_start_us":597,"threads_started":1}
I20260809 15:05:53.946808 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:53.937113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.27.186.129:40915)
0809 15:05:53.937746 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:53.937754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:53.938818 (+ 1064us) server_negotiation.cc:407] Connection header received
0809 15:05:53.938958 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:53.938964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:53.939176 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:53.939361 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:53.940202 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.941759 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.943606 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:53.945005 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:53.945207 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:53.945382 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:53.946444 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0809 15:05:53.946580 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":305,"threads_started":1}
I20260809 15:05:53.948216 24159 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae092bbb2304e6399f6b44daecd6439 (DEFAULT_TABLE table=test-table [id=ca4b1aad785b45599001487d0b1f086c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:53.949146 24363 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae092bbb2304e6399f6b44daecd6439 (DEFAULT_TABLE table=test-table [id=ca4b1aad785b45599001487d0b1f086c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:53.949705 24159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae092bbb2304e6399f6b44daecd6439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:53.950294 24363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae092bbb2304e6399f6b44daecd6439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:53.950434 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae092bbb2304e6399f6b44daecd6439 (DEFAULT_TABLE table=test-table [id=ca4b1aad785b45599001487d0b1f086c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:53.951753 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae092bbb2304e6399f6b44daecd6439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:53.961440 24501 tablet_bootstrap.cc:492] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Bootstrap starting.
I20260809 15:05:53.963559 24502 tablet_bootstrap.cc:492] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Bootstrap starting.
I20260809 15:05:53.964730 24501 tablet_bootstrap.cc:654] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:53.965186 24503 tablet_bootstrap.cc:492] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Bootstrap starting.
I20260809 15:05:53.966444 24501 log.cc:824] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:53.966971 24502 tablet_bootstrap.cc:654] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:53.967998 24502 log.cc:824] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:53.968194 24503 tablet_bootstrap.cc:654] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:53.969357 24501 tablet_bootstrap.cc:492] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: No bootstrap required, opened a new log
I20260809 15:05:53.969390 24503 log.cc:824] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:53.969623 24501 ts_tablet_manager.cc:1397] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 15:05:53.970990 24502 tablet_bootstrap.cc:492] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: No bootstrap required, opened a new log
I20260809 15:05:53.971318 24502 ts_tablet_manager.cc:1397] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260809 15:05:53.971957 24503 tablet_bootstrap.cc:492] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: No bootstrap required, opened a new log
I20260809 15:05:53.972187 24503 ts_tablet_manager.cc:1397] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260809 15:05:53.979038 24503 raft_consensus.cc:348] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:53.979351 24503 raft_consensus.cc:374] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:53.979449 24503 raft_consensus.cc:729] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c6db248ca34ac0b5ff8cc7e1373bfe, State: Initialized, Role: FOLLOWER
I20260809 15:05:53.979985 24503 consensus_queue.cc:260] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:53.980309 24501 raft_consensus.cc:348] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:53.980751 24501 raft_consensus.cc:374] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:53.980909 24501 raft_consensus.cc:729] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451648ae8b6b424d9306a3e9f3835fab, State: Initialized, Role: FOLLOWER
I20260809 15:05:53.981318 24502 raft_consensus.cc:348] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:53.981575 24503 ts_tablet_manager.cc:1428] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20260809 15:05:53.981492 24501 consensus_queue.cc:260] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:53.981740 24502 raft_consensus.cc:374] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:53.981864 24502 raft_consensus.cc:729] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e201b00dd3424bda9f6428753f774722, State: Initialized, Role: FOLLOWER
I20260809 15:05:53.982471 24502 consensus_queue.cc:260] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:53.982641 24467 heartbeater.cc:503] Master 127.27.186.190:45413 was elected leader, sending a full tablet report...
I20260809 15:05:53.983253 24501 ts_tablet_manager.cc:1428] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260809 15:05:53.984117 24263 heartbeater.cc:503] Master 127.27.186.190:45413 was elected leader, sending a full tablet report...
I20260809 15:05:53.985148 24502 ts_tablet_manager.cc:1428] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260809 15:05:54.104115 24468 tablet.cc:2369] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:05:54.104580 24045 tablet.cc:2369] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:05:54.111042 24264 tablet.cc:2369] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:05:54.113611 24508 raft_consensus.cc:482] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:54.113874 24508 raft_consensus.cc:504] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:54.115500 24508 leader_election.cc:288] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451648ae8b6b424d9306a3e9f3835fab (127.27.186.131:38641), e201b00dd3424bda9f6428753f774722 (127.27.186.130:35019)
I20260809 15:05:54.123657 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.116099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:57715 (local address 127.27.186.130:35019)
0809 15:05:54.116297 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:54.116303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:54.117315 (+ 1012us) server_negotiation.cc:407] Connection header received
0809 15:05:54.117454 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:54.117457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:54.117513 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:54.117578 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:54.118211 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.119183 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.121458 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.122639 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.122791 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.122862 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:54.123307 (+ 445us) server_negotiation.cc:299] Negotiation successful
0809 15:05:54.123421 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:05:54.123669 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.115894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:43177 (local address 127.27.186.131:38641)
0809 15:05:54.116153 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:54.116160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:54.116863 (+ 703us) server_negotiation.cc:407] Connection header received
0809 15:05:54.117070 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:54.117076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:54.117156 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:54.117237 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:54.117860 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.118930 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.121407 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.122639 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.122795 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.122861 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:54.123288 (+ 427us) server_negotiation.cc:299] Negotiation successful
0809 15:05:54.123412 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 15:05:54.123937 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.116013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:35019 (local address 127.27.186.129:57715)
0809 15:05:54.117206 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:54.117227 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:54.117342 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:54.117629 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:54.117633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:54.117660 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:54.118042 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:54.118055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.119304 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.119310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:54.121313 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:54.121325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.122890 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.122896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:54.123114 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.123171 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 15:05:54.123258 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 15:05:54.123452 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":60,"thread_start_us":898,"threads_started":1}
I20260809 15:05:54.123945 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.115929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:38641 (local address 127.27.186.129:43177)
0809 15:05:54.116753 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:54.116771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:54.116926 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:54.117295 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:54.117300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:54.117386 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:54.117721 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:54.117732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.119047 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.119051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:54.121209 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:54.121225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.122855 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.122863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:54.123105 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.123167 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 15:05:54.123260 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 15:05:54.123452 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":108,"spinlock_wait_cycles":33024,"thread_start_us":465,"threads_started":1}
I20260809 15:05:54.124943 24396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae092bbb2304e6399f6b44daecd6439" candidate_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451648ae8b6b424d9306a3e9f3835fab" is_pre_election: true
I20260809 15:05:54.124951 24193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae092bbb2304e6399f6b44daecd6439" candidate_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e201b00dd3424bda9f6428753f774722" is_pre_election: true
I20260809 15:05:54.125378 24193 raft_consensus.cc:2454] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c6db248ca34ac0b5ff8cc7e1373bfe in term 0.
I20260809 15:05:54.125387 24396 raft_consensus.cc:2454] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c6db248ca34ac0b5ff8cc7e1373bfe in term 0.
I20260809 15:05:54.125988 23849 leader_election.cc:302] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 451648ae8b6b424d9306a3e9f3835fab, b8c6db248ca34ac0b5ff8cc7e1373bfe, e201b00dd3424bda9f6428753f774722; no voters:
I20260809 15:05:54.126405 24508 raft_consensus.cc:2781] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:05:54.126545 24508 raft_consensus.cc:482] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:05:54.126636 24508 raft_consensus.cc:3037] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:54.128756 24508 raft_consensus.cc:504] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:54.129352 24508 leader_election.cc:288] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [CANDIDATE]: Term 1 election: Requested vote from peers 451648ae8b6b424d9306a3e9f3835fab (127.27.186.131:38641), e201b00dd3424bda9f6428753f774722 (127.27.186.130:35019)
I20260809 15:05:54.129688 24396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae092bbb2304e6399f6b44daecd6439" candidate_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451648ae8b6b424d9306a3e9f3835fab"
I20260809 15:05:54.129787 24193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae092bbb2304e6399f6b44daecd6439" candidate_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e201b00dd3424bda9f6428753f774722"
I20260809 15:05:54.129858 24396 raft_consensus.cc:3037] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:54.129973 24193 raft_consensus.cc:3037] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:54.131960 24396 raft_consensus.cc:2454] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c6db248ca34ac0b5ff8cc7e1373bfe in term 1.
I20260809 15:05:54.132092 24193 raft_consensus.cc:2454] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c6db248ca34ac0b5ff8cc7e1373bfe in term 1.
I20260809 15:05:54.132303 23851 leader_election.cc:302] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451648ae8b6b424d9306a3e9f3835fab, b8c6db248ca34ac0b5ff8cc7e1373bfe; no voters:
I20260809 15:05:54.132555 24508 raft_consensus.cc:2781] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:05:54.132984 24508 raft_consensus.cc:686] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 1 LEADER]: Becoming Leader. State: Replica: b8c6db248ca34ac0b5ff8cc7e1373bfe, State: Running, Role: LEADER
I20260809 15:05:54.133448 24508 consensus_queue.cc:237] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } }
I20260809 15:05:54.137763 23707 catalog_manager.cc:5495] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe reported cstate change: term changed from 0 to 1, leader changed from <none> to b8c6db248ca34ac0b5ff8cc7e1373bfe (127.27.186.129). New cstate: current_term: 1 leader_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8c6db248ca34ac0b5ff8cc7e1373bfe" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 40915 } health_report { overall_health: HEALTHY } } }
I20260809 15:05:54.166740 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.160786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.27.186.129:40915)
0809 15:05:54.160975 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:54.160981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:54.161108 (+ 127us) server_negotiation.cc:407] Connection header received
0809 15:05:54.161311 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:54.161315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:54.161397 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:54.161509 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:05:54.161789 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.162569 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.164095 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.164714 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.164836 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.165477 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:05:54.166130 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:05:54.166275 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:54.166371 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 15:05:54.166452 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:05:54.173992 24396 raft_consensus.cc:1261] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 1 FOLLOWER]: Refusing update from remote peer b8c6db248ca34ac0b5ff8cc7e1373bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:05:54.173990 24193 raft_consensus.cc:1261] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 1 FOLLOWER]: Refusing update from remote peer b8c6db248ca34ac0b5ff8cc7e1373bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:05:54.174873 24514 consensus_queue.cc:1035] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "451648ae8b6b424d9306a3e9f3835fab" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:05:54.175203 24508 consensus_queue.cc:1035] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e201b00dd3424bda9f6428753f774722" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:05:54.185217 24520 mvcc.cc:205] Tried to move back new op lower bound from 7316635460287901696 to 7316635460134080512. Current Snapshot: MvccSnapshot[applied={T|T < 7316635460287901696}]
I20260809 15:05:54.201256 23707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:05:54.201530 23707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:05:54.204981 23707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6399c74ab8847eaaa9b30f37f7c89fe: Sending DeleteTablet for 3 replicas of tablet 4ae092bbb2304e6399f6b44daecd6439
I20260809 15:05:54.206050 24363 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae092bbb2304e6399f6b44daecd6439 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:54 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:56144
I20260809 15:05:54.206161 24159 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae092bbb2304e6399f6b44daecd6439 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:54 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:47892
I20260809 15:05:54.206847 23923 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae092bbb2304e6399f6b44daecd6439 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:54 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:49486
I20260809 15:05:54.207592 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 23819
I20260809 15:05:54.219115 24542 tablet_replica.cc:330] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: stopping tablet replica
I20260809 15:05:54.219378 24545 tablet_replica.cc:330] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: stopping tablet replica
I20260809 15:05:54.219520 24542 raft_consensus.cc:2229] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:05:54.219985 24542 raft_consensus.cc:2258] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:54.220168 24545 raft_consensus.cc:2229] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:05:54.220914 24545 raft_consensus.cc:2258] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:54.222273 24542 ts_tablet_manager.cc:1905] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:54.223306 24544 tablet_replica.cc:330] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: stopping tablet replica
I20260809 15:05:54.223547 24545 ts_tablet_manager.cc:1905] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:54.226384 24544 raft_consensus.cc:2229] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:05:54.227035 24544 raft_consensus.cc:2258] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:54.229439 24542 ts_tablet_manager.cc:1918] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:54.229640 24542 log.cc:1196] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/4ae092bbb2304e6399f6b44daecd6439
I20260809 15:05:54.230312 24542 ts_tablet_manager.cc:1939] T 4ae092bbb2304e6399f6b44daecd6439 P 451648ae8b6b424d9306a3e9f3835fab: Deleting consensus metadata
I20260809 15:05:54.232076 23685 catalog_manager.cc:4841] TS 451648ae8b6b424d9306a3e9f3835fab (127.27.186.131:38641): tablet 4ae092bbb2304e6399f6b44daecd6439 (table test-table [id=ca4b1aad785b45599001487d0b1f086c]) successfully deleted
I20260809 15:05:54.232926 24544 ts_tablet_manager.cc:1905] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 15:05:54 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:05:54.240633 24545 ts_tablet_manager.cc:1918] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:54.240906 24545 log.cc:1196] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/4ae092bbb2304e6399f6b44daecd6439
I20260809 15:05:54.241566 24545 ts_tablet_manager.cc:1939] T 4ae092bbb2304e6399f6b44daecd6439 P b8c6db248ca34ac0b5ff8cc7e1373bfe: Deleting consensus metadata
I20260809 15:05:54.243333 24544 ts_tablet_manager.cc:1918] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:54.243541 24544 log.cc:1196] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/4ae092bbb2304e6399f6b44daecd6439
I20260809 15:05:54.244148 24544 ts_tablet_manager.cc:1939] T 4ae092bbb2304e6399f6b44daecd6439 P e201b00dd3424bda9f6428753f774722: Deleting consensus metadata
I20260809 15:05:54.244814 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.208903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.27.186.129:40915)
0809 15:05:54.209064 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:54.209071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:54.215346 (+ 6275us) server_negotiation.cc:407] Connection header received
0809 15:05:54.215985 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:54.216017 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:54.216102 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:54.216233 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:54.222251 (+ 6018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.223753 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.225548 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.226274 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.226367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.237103 (+ 10736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:54.237118 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:54.237134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:54.237185 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:54.241593 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:54.242477 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:54.242487 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:54.242491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:54.242678 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:54.243106 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:54.243112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:54.243115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:54.243589 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:54.243896 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:54.244208 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 15:05:54.244457 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:05:54.245639 24034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:54.246186 23684 catalog_manager.cc:4841] TS b8c6db248ca34ac0b5ff8cc7e1373bfe (127.27.186.129:40915): tablet 4ae092bbb2304e6399f6b44daecd6439 (table test-table [id=ca4b1aad785b45599001487d0b1f086c]) successfully deleted
I20260809 15:05:54.247387 23684 catalog_manager.cc:4841] TS e201b00dd3424bda9f6428753f774722 (127.27.186.130:35019): tablet 4ae092bbb2304e6399f6b44daecd6439 (table test-table [id=ca4b1aad785b45599001487d0b1f086c]) successfully deleted
I20260809 15:05:54.361663 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 23819
I20260809 15:05:54.389571 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 24054
Aug 09 15:05:54 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:05:54.407297 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.390389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.27.186.130:35019)
0809 15:05:54.390575 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:54.390580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:54.390599 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:05:54.390750 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:54.390754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:54.390824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:54.390904 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:54.392062 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.392781 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.394142 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.394537 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.394637 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.402170 (+ 7533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:54.402182 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:54.402196 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:54.402229 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:54.404936 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:54.405582 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:54.405589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:54.405591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:54.405718 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:54.405999 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:54.406002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:54.406004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:54.406440 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:54.406676 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:54.406882 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 15:05:54.407037 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:05:54.407898 24256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:54.494095 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 24054
I20260809 15:05:54.517639 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 24268
Aug 09 15:05:54 dist-test-slave-q9h9 krb5kdc[23636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:05:54.531144 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:54.518353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.27.186.131:38641)
0809 15:05:54.518543 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:54.518548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:54.518567 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:05:54.518716 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:54.518720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:54.518800 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:54.518878 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:54.520018 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.520710 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.521845 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:54.522248 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:54.522333 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:54.526404 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:54.526420 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:54.526437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:54.526469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:54.528906 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:54.529465 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:54.529471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:54.529474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:54.529594 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:54.529863 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:54.529866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:54.529868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:54.530253 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:54.530466 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:54.530679 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 15:05:54.530843 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 15:05:54.531718 24458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:54.616230 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 24268
I20260809 15:05:54.639091 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 23653
I20260809 15:05:54.640247 23765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:54.744836 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 23653
2026-08-09T15:05:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4331 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T15:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:05:54Z Disabled control of system clock
I20260809 15:05:54.829896 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38761
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 15:05:55.142109 24559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:55.142371 24559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:55.142488 24559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:55.149474 24559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:05:55.149564 24559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:55.149626 24559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:55.149708 24559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:05:55.149760 24559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:05:55.158497 24559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:38761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:55.159569 24559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:55.161127 24559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:55.172545 24566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:55.172614 24568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:55.172576 24570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:55.173563 24559 server_base.cc:1034] running on GCE node
I20260809 15:05:55.174285 24559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:55.175518 24559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:55.176723 24559 hybrid_clock.cc:648] HybridClock initialized: now 1786287955176669 us; error 41 us; skew 500 ppm
I20260809 15:05:55.177024 24559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:05:55.182704 24559 webserver.cc:460] Webserver started at http://127.27.186.190:39429/ using document root <none> and password file <none>
I20260809 15:05:55.183302 24559 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:55.183398 24559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:55.183694 24559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:55.186425 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "736f90c0927448c7aaadd19cd24e0738"
format_stamp: "Formatted at 2026-08-09 15:05:55 on dist-test-slave-q9h9"
server_key: "8d6990bfb53cc484ac6f2d19a27f711f"
server_key_iv: "c35e3bb6b37613cd64b06b4a2bda57be"
server_key_version: "encryptionkey@0"
I20260809 15:05:55.187170 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "736f90c0927448c7aaadd19cd24e0738"
format_stamp: "Formatted at 2026-08-09 15:05:55 on dist-test-slave-q9h9"
server_key: "8d6990bfb53cc484ac6f2d19a27f711f"
server_key_iv: "c35e3bb6b37613cd64b06b4a2bda57be"
server_key_version: "encryptionkey@0"
I20260809 15:05:55.191903 24559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 15:05:55.195045 24580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:55.196208 24559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:05:55.196367 24559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "736f90c0927448c7aaadd19cd24e0738"
format_stamp: "Formatted at 2026-08-09 15:05:55 on dist-test-slave-q9h9"
server_key: "8d6990bfb53cc484ac6f2d19a27f711f"
server_key_iv: "c35e3bb6b37613cd64b06b4a2bda57be"
server_key_version: "encryptionkey@0"
I20260809 15:05:55.196554 24559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:55.221089 24559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:55.221761 24559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:55.222008 24559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:55.238277 24559 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:38761
I20260809 15:05:55.238315 24661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:38761 every 8 connection(s)
I20260809 15:05:55.239401 24559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:05:55.239986 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 24559
I20260809 15:05:55.240149 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:05:55.240726 28394 external_mini_cluster.cc:1428] Setting key a743ba959f16eeae8645073388555b35
I20260809 15:05:55.243357 24666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:55.253418 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:55.242496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.27.186.190:38761)
0809 15:05:55.243373 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:55.243383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:55.243420 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:05:55.244971 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:55.244989 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:55.245391 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:55.245695 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:55.246295 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:55.247566 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:55.249448 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:55.250116 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:55.250245 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:55.251358 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:55.251368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:55.251379 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:55.251872 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:55.251950 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:55.251969 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:55.252053 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:55.252283 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 15:05:55.252565 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":326,"threads_started":1}
I20260809 15:05:55.254908 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738: Bootstrap starting.
I20260809 15:05:55.257995 24666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:55.259250 24666 log.cc:824] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:55.261850 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738: No bootstrap required, opened a new log
I20260809 15:05:55.268841 24666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f90c0927448c7aaadd19cd24e0738" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38761 } }
I20260809 15:05:55.269141 24666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:55.269243 24666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736f90c0927448c7aaadd19cd24e0738, State: Initialized, Role: FOLLOWER
I20260809 15:05:55.269742 24666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f90c0927448c7aaadd19cd24e0738" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38761 } }
I20260809 15:05:55.269948 24666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:05:55.270090 24666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:05:55.270251 24666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:55.272239 24666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f90c0927448c7aaadd19cd24e0738" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38761 } }
I20260809 15:05:55.272671 24666 leader_election.cc:302] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736f90c0927448c7aaadd19cd24e0738; no voters:
I20260809 15:05:55.273218 24666 leader_election.cc:288] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:05:55.273367 24675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:05:55.274080 24675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [term 1 LEADER]: Becoming Leader. State: Replica: 736f90c0927448c7aaadd19cd24e0738, State: Running, Role: LEADER
I20260809 15:05:55.274586 24675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f90c0927448c7aaadd19cd24e0738" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38761 } }
I20260809 15:05:55.275141 24666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:05:55.277272 24676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736f90c0927448c7aaadd19cd24e0738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f90c0927448c7aaadd19cd24e0738" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38761 } } }
I20260809 15:05:55.277854 24677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736f90c0927448c7aaadd19cd24e0738. Latest consensus state: current_term: 1 leader_uuid: "736f90c0927448c7aaadd19cd24e0738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f90c0927448c7aaadd19cd24e0738" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38761 } } }
I20260809 15:05:55.278182 24676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:55.278288 24677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:55.279402 24690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:05:55.283459 24690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:05:55.291270 24690 catalog_manager.cc:1347] Generated new cluster ID: 96096a7c3ad14b9fa8f1d35cf4f1490e
I20260809 15:05:55.291363 24690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:05:55.300506 24690 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:05:55.301729 24690 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:05:55.308271 24690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738: Generated new TSK 0
I20260809 15:05:55.308892 24690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:05:55.317957 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38761
--builtin_ntp_servers=127.27.186.148:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 15:05:55.666638 24704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:55.666888 24704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:55.666997 24704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:55.673913 24704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:55.673997 24704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:55.674152 24704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:05:55.682940 24704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:55.684057 24704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:55.685752 24704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:55.697908 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:55.698210 24714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:55.698104 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:55.699252 24704 server_base.cc:1034] running on GCE node
I20260809 15:05:55.699821 24704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:55.700670 24704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:55.701871 24704 hybrid_clock.cc:648] HybridClock initialized: now 1786287955701810 us; error 50 us; skew 500 ppm
I20260809 15:05:55.702200 24704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:05:55.704262 24704 webserver.cc:460] Webserver started at http://127.27.186.129:37945/ using document root <none> and password file <none>
I20260809 15:05:55.704867 24704 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:55.704970 24704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:55.705267 24704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:55.707962 24704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "a53a1319fca841d5ac72364860da4ea8"
format_stamp: "Formatted at 2026-08-09 15:05:55 on dist-test-slave-q9h9"
server_key: "1dcae89d879c3674210463d67d90d401"
server_key_iv: "01347f4cca3fc5bf66206452324f366b"
server_key_version: "encryptionkey@0"
I20260809 15:05:55.708711 24704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "a53a1319fca841d5ac72364860da4ea8"
format_stamp: "Formatted at 2026-08-09 15:05:55 on dist-test-slave-q9h9"
server_key: "1dcae89d879c3674210463d67d90d401"
server_key_iv: "01347f4cca3fc5bf66206452324f366b"
server_key_version: "encryptionkey@0"
I20260809 15:05:55.713394 24704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:05:55.716693 24729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:55.717834 24704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:05:55.717974 24704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "a53a1319fca841d5ac72364860da4ea8"
format_stamp: "Formatted at 2026-08-09 15:05:55 on dist-test-slave-q9h9"
server_key: "1dcae89d879c3674210463d67d90d401"
server_key_iv: "01347f4cca3fc5bf66206452324f366b"
server_key_version: "encryptionkey@0"
I20260809 15:05:55.718196 24704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:55.743008 24704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:55.743692 24704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:55.743954 24704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:55.744853 24704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:55.746161 24704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:55.746263 24704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:55.746359 24704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:55.746433 24704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:55.776633 24704 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:32775
I20260809 15:05:55.776682 24937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:32775 every 8 connection(s)
I20260809 15:05:55.777805 24704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:05:55.780403 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 24704
I20260809 15:05:55.780578 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:05:55.781028 28394 external_mini_cluster.cc:1428] Setting key 37e0c2b7adb61c5e0b2e49fc57bafe2b
I20260809 15:05:55.783675 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38761
--builtin_ntp_servers=127.27.186.148:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 15:05:55.789772 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:55.780265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:59055 (local address 127.27.186.190:38761)
0809 15:05:55.780781 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:55.780789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:55.782038 (+ 1249us) server_negotiation.cc:407] Connection header received
0809 15:05:55.783801 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:55.783807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:55.783885 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:55.783986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:55.784995 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:55.785701 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:55.787230 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:55.787665 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:55.787777 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:55.788994 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:55.789000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:55.789003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:55.789019 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:55.789057 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:55.789069 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:55.789153 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:55.789357 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 15:05:55.789449 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260809 15:05:55.790709 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:55.780849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:38761 (local address 127.27.186.129:59055)
0809 15:05:55.781872 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:55.781923 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:55.783543 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:55.784215 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:55.784223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:55.784309 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:55.784834 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:55.784850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:55.785856 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:55.785860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:55.787089 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:55.787104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:55.787883 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:55.787891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:55.788103 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:55.788708 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:05:55.788711 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:55.788746 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:05:55.788819 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:55.789167 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:55.789179 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:55.789186 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:05:55.789331 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 15:05:55.789696 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":335,"threads_started":1}
I20260809 15:05:55.792058 24938 heartbeater.cc:347] Connected to a master server at 127.27.186.190:38761
I20260809 15:05:55.792333 24938 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:55.793010 24938 heartbeater.cc:511] Master 127.27.186.190:38761 requested a full tablet report, sending...
I20260809 15:05:55.794981 24600 ts_manager.cc:194] Registered new tserver with Master: a53a1319fca841d5ac72364860da4ea8 (127.27.186.129:32775)
I20260809 15:05:55.796643 24600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.129:59055
W20260809 15:05:56.102103 24944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:56.102357 24944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:56.102484 24944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:56.109295 24944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:56.109390 24944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:56.109517 24944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:05:56.118052 24944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:56.119104 24944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:56.120685 24944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:56.132864 24961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:56.132949 24951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:56.132885 24955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:56.133711 24944 server_base.cc:1034] running on GCE node
I20260809 15:05:56.134321 24944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:56.135169 24944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:56.136371 24944 hybrid_clock.cc:648] HybridClock initialized: now 1786287956136318 us; error 46 us; skew 500 ppm
I20260809 15:05:56.136647 24944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:05:56.138662 24944 webserver.cc:460] Webserver started at http://127.27.186.130:45355/ using document root <none> and password file <none>
I20260809 15:05:56.139250 24944 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:56.139339 24944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:56.139575 24944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:56.142272 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "ffa9c1c644e141689911df35cdcb0e2c"
format_stamp: "Formatted at 2026-08-09 15:05:56 on dist-test-slave-q9h9"
server_key: "134182ea69377066e455f11d3d1baca4"
server_key_iv: "bcf5de86d55718316b91f29e197dfb38"
server_key_version: "encryptionkey@0"
I20260809 15:05:56.143009 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "ffa9c1c644e141689911df35cdcb0e2c"
format_stamp: "Formatted at 2026-08-09 15:05:56 on dist-test-slave-q9h9"
server_key: "134182ea69377066e455f11d3d1baca4"
server_key_iv: "bcf5de86d55718316b91f29e197dfb38"
server_key_version: "encryptionkey@0"
I20260809 15:05:56.147619 24944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:05:56.150817 24969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:56.151965 24944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:05:56.152103 24944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "ffa9c1c644e141689911df35cdcb0e2c"
format_stamp: "Formatted at 2026-08-09 15:05:56 on dist-test-slave-q9h9"
server_key: "134182ea69377066e455f11d3d1baca4"
server_key_iv: "bcf5de86d55718316b91f29e197dfb38"
server_key_version: "encryptionkey@0"
I20260809 15:05:56.152292 24944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:56.166744 24944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:56.167411 24944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:56.167665 24944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:56.168576 24944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:56.169898 24944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:56.170006 24944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:56.170127 24944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:56.170204 24944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:56.200644 24944 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:44587
I20260809 15:05:56.200713 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:44587 every 8 connection(s)
I20260809 15:05:56.201869 24944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:05:56.210851 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 24944
I20260809 15:05:56.211007 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:05:56.211418 28394 external_mini_cluster.cc:1428] Setting key 396ba8c0431d5a4cce7fdb371731868e
I20260809 15:05:56.213272 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.204078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:47793 (local address 127.27.186.190:38761)
0809 15:05:56.204256 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.204261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.205429 (+ 1168us) server_negotiation.cc:407] Connection header received
0809 15:05:56.207337 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.207342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.207409 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.207492 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:56.208389 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.209081 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.210745 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.211184 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.211301 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.212504 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:56.212508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:56.212512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:56.212528 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:56.212560 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:56.212572 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:56.212641 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.212853 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.212930 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 15:05:56.214195 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.204480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:38761 (local address 127.27.186.130:47793)
0809 15:05:56.205280 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.205313 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.207023 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.207679 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.207686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.207762 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:56.208264 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.208280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.209209 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.209212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.210591 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.210612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.211391 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.211398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.211633 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.212224 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:05:56.212227 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:56.212261 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:05:56.212333 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:56.212662 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:56.212673 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:56.212680 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.212853 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.213199 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":248,"threads_started":1}
I20260809 15:05:56.214449 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38761
--builtin_ntp_servers=127.27.186.148:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 15:05:56.215430 25145 heartbeater.cc:347] Connected to a master server at 127.27.186.190:38761
I20260809 15:05:56.215726 25145 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:56.216549 25145 heartbeater.cc:511] Master 127.27.186.190:38761 requested a full tablet report, sending...
I20260809 15:05:56.217883 24600 ts_manager.cc:194] Registered new tserver with Master: ffa9c1c644e141689911df35cdcb0e2c (127.27.186.130:44587)
I20260809 15:05:56.218822 24600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.130:47793
W20260809 15:05:56.537568 25149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:56.537793 25149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:56.537910 25149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:56.544790 25149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:56.544868 25149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:56.545014 25149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:05:56.553802 25149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:56.554857 25149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:56.556396 25149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:56.567940 25160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:56.568043 25164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:56.568007 25158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:56.568759 25149 server_base.cc:1034] running on GCE node
I20260809 15:05:56.569360 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:56.570173 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:56.571372 25149 hybrid_clock.cc:648] HybridClock initialized: now 1786287956571329 us; error 44 us; skew 500 ppm
I20260809 15:05:56.571645 25149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:05:56.573589 25149 webserver.cc:460] Webserver started at http://127.27.186.131:45049/ using document root <none> and password file <none>
I20260809 15:05:56.574247 25149 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:56.574347 25149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:56.574623 25149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:56.577203 25149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "079c642f733040db97f65f7ec699dd3a"
format_stamp: "Formatted at 2026-08-09 15:05:56 on dist-test-slave-q9h9"
server_key: "ce903d310160f0cf06c4b75873747388"
server_key_iv: "a9e93b6afd8cf8606c1aef14fe6dc220"
server_key_version: "encryptionkey@0"
I20260809 15:05:56.577965 25149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "079c642f733040db97f65f7ec699dd3a"
format_stamp: "Formatted at 2026-08-09 15:05:56 on dist-test-slave-q9h9"
server_key: "ce903d310160f0cf06c4b75873747388"
server_key_iv: "a9e93b6afd8cf8606c1aef14fe6dc220"
server_key_version: "encryptionkey@0"
I20260809 15:05:56.582561 25149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 15:05:56.585667 25173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:56.586833 25149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:05:56.586972 25149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "079c642f733040db97f65f7ec699dd3a"
format_stamp: "Formatted at 2026-08-09 15:05:56 on dist-test-slave-q9h9"
server_key: "ce903d310160f0cf06c4b75873747388"
server_key_iv: "a9e93b6afd8cf8606c1aef14fe6dc220"
server_key_version: "encryptionkey@0"
I20260809 15:05:56.587154 25149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:56.600220 25149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:56.600904 25149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:56.601166 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:56.601981 25149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:56.603255 25149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:56.603350 25149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:56.603457 25149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:56.603528 25149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:56.641490 25149 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:34629
I20260809 15:05:56.641569 25338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:34629 every 8 connection(s)
I20260809 15:05:56.642974 25149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:05:56.651192 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 25149
I20260809 15:05:56.651345 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:05:56.651763 28394 external_mini_cluster.cc:1428] Setting key e4ba171b2b4adae52cee9d72595e59a2
I20260809 15:05:56.657588 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.645514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:40321 (local address 127.27.186.190:38761)
0809 15:05:56.645702 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.645708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.647349 (+ 1641us) server_negotiation.cc:407] Connection header received
0809 15:05:56.649877 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.649882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.649958 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.650042 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:56.652088 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.653047 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.654903 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.655469 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.655590 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.656829 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:56.656834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:56.656838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:56.656853 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:56.656883 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:56.656894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:56.656963 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.657182 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.657284 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:05:56.658895 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.646161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:38761 (local address 127.27.186.131:40321)
0809 15:05:56.647194 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.647238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.649536 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.651253 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.651264 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.651375 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:56.651921 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.651938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.653189 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.653194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.654747 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.654762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.655663 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.655670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.655918 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.656537 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:05:56.656539 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:56.656573 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:05:56.656641 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:56.656978 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:56.656990 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:56.656998 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.657138 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.657472 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":309,"threads_started":1}
I20260809 15:05:56.660214 25339 heartbeater.cc:347] Connected to a master server at 127.27.186.190:38761
I20260809 15:05:56.660492 25339 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:56.661183 25339 heartbeater.cc:511] Master 127.27.186.190:38761 requested a full tablet report, sending...
I20260809 15:05:56.662554 24600 ts_manager.cc:194] Registered new tserver with Master: 079c642f733040db97f65f7ec699dd3a (127.27.186.131:34629)
I20260809 15:05:56.663467 24600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.131:40321
I20260809 15:05:56.666750 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:05:56.675406 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.670469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.27.186.190:38761)
0809 15:05:56.670644 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.670650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.670997 (+ 347us) server_negotiation.cc:407] Connection header received
0809 15:05:56.671113 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.671117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.671179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.671258 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:56.671637 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.672260 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.673509 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.673874 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.673956 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.674848 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:56.674853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:56.674856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:56.674871 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:56.674897 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:56.674908 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:56.674973 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.675077 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.675141 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:05:56.678609 24600 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:05:56.684983 24600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:05:56.705001 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.695766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:44587 (local address 127.0.0.1:52840)
0809 15:05:56.697435 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.697450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.697552 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.698534 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.698539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.698571 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:56.698937 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.698948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.700596 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.700601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.702181 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.702193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.704148 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.704152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.704293 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.704325 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.704439 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.704612 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":40,"spinlock_wait_cycles":4992,"thread_start_us":1369,"threads_started":1}
I20260809 15:05:56.705314 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.695829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.27.186.129:32775)
0809 15:05:56.696480 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.696488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.697291 (+ 803us) server_negotiation.cc:407] Connection header received
0809 15:05:56.697393 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.697400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.697631 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.697807 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:56.698518 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.699947 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.702767 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.704307 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.704443 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.704637 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.704833 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.704996 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":345,"threads_started":1}
I20260809 15:05:56.705601 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.696481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.27.186.130:44587)
0809 15:05:56.697680 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.697697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.697724 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:05:56.697822 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.697829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.698063 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.698257 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:56.699116 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.700475 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.702382 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.703702 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.703888 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.704040 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.705115 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.705265 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":1552,"threads_started":1}
I20260809 15:05:56.706060 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.695597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:32775 (local address 127.0.0.1:52286)
0809 15:05:56.696867 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.696905 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.697088 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.697829 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.697836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.697862 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:56.698279 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.698294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.700201 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.700206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.702567 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.702584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.704483 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.704490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.704667 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.704694 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.704768 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.704885 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":22,"thread_start_us":902,"threads_started":1}
I20260809 15:05:56.706789 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.697521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34629 (local address 127.0.0.1:52554)
0809 15:05:56.698574 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.698591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.698701 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.699765 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.699770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.699793 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:56.700192 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.700203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.702332 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.702337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.704200 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.704220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.706166 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.706172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.706332 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.706362 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.706456 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.706563 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":814,"threads_started":1}
I20260809 15:05:56.707191 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.698047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.27.186.131:34629)
0809 15:05:56.699220 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.699228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.699256 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:05:56.699331 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.699338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.699520 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.700399 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:56.700644 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.702198 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.704414 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.705692 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.705915 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.706134 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.706504 (+ 370us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.706652 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":875,"threads_started":1}
I20260809 15:05:56.709281 25052 tablet_service.cc:1467] Processing CreateTablet for tablet ad361cfefef24285887ae1fa010c41ba (DEFAULT_TABLE table=test-table [id=f5a9b17872c14201ab704223658a2b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:56.709283 24806 tablet_service.cc:1467] Processing CreateTablet for tablet ad361cfefef24285887ae1fa010c41ba (DEFAULT_TABLE table=test-table [id=f5a9b17872c14201ab704223658a2b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:56.710439 25052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad361cfefef24285887ae1fa010c41ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:56.710508 24806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad361cfefef24285887ae1fa010c41ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:56.712956 25248 tablet_service.cc:1467] Processing CreateTablet for tablet ad361cfefef24285887ae1fa010c41ba (DEFAULT_TABLE table=test-table [id=f5a9b17872c14201ab704223658a2b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:05:56.714244 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad361cfefef24285887ae1fa010c41ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:56.721217 25365 tablet_bootstrap.cc:492] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Bootstrap starting.
I20260809 15:05:56.724112 25365 tablet_bootstrap.cc:654] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:56.724915 25367 tablet_bootstrap.cc:492] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Bootstrap starting.
I20260809 15:05:56.726337 25365 log.cc:824] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:56.727284 25368 tablet_bootstrap.cc:492] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Bootstrap starting.
I20260809 15:05:56.728669 25367 tablet_bootstrap.cc:654] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:56.729818 25367 log.cc:824] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:56.730494 25368 tablet_bootstrap.cc:654] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:56.730556 25365 tablet_bootstrap.cc:492] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: No bootstrap required, opened a new log
I20260809 15:05:56.730784 25365 ts_tablet_manager.cc:1397] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 15:05:56.731485 25368 log.cc:824] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:56.732677 25367 tablet_bootstrap.cc:492] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: No bootstrap required, opened a new log
I20260809 15:05:56.732933 25367 ts_tablet_manager.cc:1397] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 15:05:56.734076 25368 tablet_bootstrap.cc:492] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: No bootstrap required, opened a new log
I20260809 15:05:56.734311 25368 ts_tablet_manager.cc:1397] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:05:56.740787 25368 raft_consensus.cc:348] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.741104 25368 raft_consensus.cc:374] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:56.740882 25365 raft_consensus.cc:348] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.741199 25368 raft_consensus.cc:729] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079c642f733040db97f65f7ec699dd3a, State: Initialized, Role: FOLLOWER
I20260809 15:05:56.741271 25365 raft_consensus.cc:374] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:56.741372 25365 raft_consensus.cc:729] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa9c1c644e141689911df35cdcb0e2c, State: Initialized, Role: FOLLOWER
I20260809 15:05:56.741780 25368 consensus_queue.cc:260] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.741999 25365 consensus_queue.cc:260] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.742864 25339 heartbeater.cc:503] Master 127.27.186.190:38761 was elected leader, sending a full tablet report...
I20260809 15:05:56.742750 25367 raft_consensus.cc:348] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.743144 25367 raft_consensus.cc:374] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:56.743244 25367 raft_consensus.cc:729] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53a1319fca841d5ac72364860da4ea8, State: Initialized, Role: FOLLOWER
I20260809 15:05:56.743363 25368 ts_tablet_manager.cc:1428] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260809 15:05:56.743774 25367 consensus_queue.cc:260] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.746622 24938 heartbeater.cc:503] Master 127.27.186.190:38761 was elected leader, sending a full tablet report...
I20260809 15:05:56.748451 25367 ts_tablet_manager.cc:1428] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260809 15:05:56.751173 25145 heartbeater.cc:503] Master 127.27.186.190:38761 was elected leader, sending a full tablet report...
I20260809 15:05:56.751770 25365 ts_tablet_manager.cc:1428] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Time spent starting tablet: real 0.021s user 0.005s sys 0.013s
W20260809 15:05:56.779827 24940 tablet.cc:2369] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:05:56.827271 25374 raft_consensus.cc:482] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:05:56.827474 25374 raft_consensus.cc:504] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.828858 25374 leader_election.cc:288] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a53a1319fca841d5ac72364860da4ea8 (127.27.186.129:32775), ffa9c1c644e141689911df35cdcb0e2c (127.27.186.130:44587)
I20260809 15:05:56.834769 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.829272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:39327 (local address 127.27.186.129:32775)
0809 15:05:56.829449 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.829455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.829549 (+ 94us) server_negotiation.cc:407] Connection header received
0809 15:05:56.829728 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.829733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.829817 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.829913 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:56.830437 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.831280 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.833014 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.833930 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.834069 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.834119 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.834427 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.834531 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:05:56.834802 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.829107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:32775 (local address 127.27.186.131:39327)
0809 15:05:56.829439 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.829458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.829595 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.829989 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.829993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.830107 (+ 114us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:56.830295 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.830307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.831408 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.831413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.832859 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.832871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.834101 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.834107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.834270 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.834319 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.834413 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.834554 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"spinlock_wait_cycles":93952}
I20260809 15:05:56.835299 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.829210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:53553 (local address 127.27.186.130:44587)
0809 15:05:56.829454 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.829460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.830003 (+ 543us) server_negotiation.cc:407] Connection header received
0809 15:05:56.830221 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.830226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.830292 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.830385 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:05:56.830946 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.831722 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.833516 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.834410 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.834542 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.834603 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.834963 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.835079 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260809 15:05:56.835748 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.829212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:44587 (local address 127.27.186.131:53553)
0809 15:05:56.829906 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:56.829923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:56.830101 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:56.830402 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:56.830406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:56.830430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:05:56.830790 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.830803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.831878 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.831882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.833327 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:56.833341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.834564 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.834569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:56.834751 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.834784 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:05:56.835454 (+ 670us) client_negotiation.cc:240] Negotiation successful
0809 15:05:56.835547 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":391,"threads_started":1}
I20260809 15:05:56.835737 24846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad361cfefef24285887ae1fa010c41ba" candidate_uuid: "079c642f733040db97f65f7ec699dd3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53a1319fca841d5ac72364860da4ea8" is_pre_election: true
I20260809 15:05:56.836197 24846 raft_consensus.cc:2454] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 079c642f733040db97f65f7ec699dd3a in term 0.
I20260809 15:05:56.836642 25082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad361cfefef24285887ae1fa010c41ba" candidate_uuid: "079c642f733040db97f65f7ec699dd3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa9c1c644e141689911df35cdcb0e2c" is_pre_election: true
I20260809 15:05:56.836838 25175 leader_election.cc:302] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 079c642f733040db97f65f7ec699dd3a, a53a1319fca841d5ac72364860da4ea8; no voters:
I20260809 15:05:56.837044 25082 raft_consensus.cc:2454] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 079c642f733040db97f65f7ec699dd3a in term 0.
I20260809 15:05:56.837175 25374 raft_consensus.cc:2781] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:05:56.837311 25374 raft_consensus.cc:482] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:05:56.837417 25374 raft_consensus.cc:3037] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:56.839530 25374 raft_consensus.cc:504] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.840031 25374 leader_election.cc:288] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [CANDIDATE]: Term 1 election: Requested vote from peers a53a1319fca841d5ac72364860da4ea8 (127.27.186.129:32775), ffa9c1c644e141689911df35cdcb0e2c (127.27.186.130:44587)
I20260809 15:05:56.840413 25082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad361cfefef24285887ae1fa010c41ba" candidate_uuid: "079c642f733040db97f65f7ec699dd3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa9c1c644e141689911df35cdcb0e2c"
I20260809 15:05:56.840448 24846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad361cfefef24285887ae1fa010c41ba" candidate_uuid: "079c642f733040db97f65f7ec699dd3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53a1319fca841d5ac72364860da4ea8"
I20260809 15:05:56.840564 25082 raft_consensus.cc:3037] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:56.840601 24846 raft_consensus.cc:3037] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:56.842701 24846 raft_consensus.cc:2454] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 079c642f733040db97f65f7ec699dd3a in term 1.
I20260809 15:05:56.842779 25082 raft_consensus.cc:2454] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 079c642f733040db97f65f7ec699dd3a in term 1.
I20260809 15:05:56.843070 25175 leader_election.cc:302] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 079c642f733040db97f65f7ec699dd3a, a53a1319fca841d5ac72364860da4ea8; no voters:
I20260809 15:05:56.843319 25374 raft_consensus.cc:2781] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:05:56.843686 25374 raft_consensus.cc:686] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 1 LEADER]: Becoming Leader. State: Replica: 079c642f733040db97f65f7ec699dd3a, State: Running, Role: LEADER
I20260809 15:05:56.844142 25374 consensus_queue.cc:237] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } }
I20260809 15:05:56.848328 24600 catalog_manager.cc:5495] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 079c642f733040db97f65f7ec699dd3a (127.27.186.131). New cstate: current_term: 1 leader_uuid: "079c642f733040db97f65f7ec699dd3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "079c642f733040db97f65f7ec699dd3a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34629 } health_report { overall_health: HEALTHY } } }
I20260809 15:05:56.891695 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.886554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.27.186.131:34629)
0809 15:05:56.886715 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.886721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.886870 (+ 149us) server_negotiation.cc:407] Connection header received
0809 15:05:56.887055 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.887060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.887127 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.887243 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:05:56.887603 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.888291 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.889713 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.890188 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.890308 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.890778 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:05:56.891173 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:05:56.891306 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.891419 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.891487 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 15:05:56.894613 25340 tablet.cc:2369] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:05:56.898710 25082 raft_consensus.cc:1261] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 1 FOLLOWER]: Refusing update from remote peer 079c642f733040db97f65f7ec699dd3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:05:56.898763 24846 raft_consensus.cc:1261] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 1 FOLLOWER]: Refusing update from remote peer 079c642f733040db97f65f7ec699dd3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:05:56.899480 25383 consensus_queue.cc:1035] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa9c1c644e141689911df35cdcb0e2c" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:05:56.899803 25374 consensus_queue.cc:1035] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53a1319fca841d5ac72364860da4ea8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:05:56.922389 24598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:05:56.922690 24598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:05:56.926254 24598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 736f90c0927448c7aaadd19cd24e0738: Sending DeleteTablet for 3 replicas of tablet ad361cfefef24285887ae1fa010c41ba
I20260809 15:05:56.927243 24806 tablet_service.cc:1514] Processing DeleteTablet for tablet ad361cfefef24285887ae1fa010c41ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:56 UTC) from {username='slave'} at 127.0.0.1:52286
I20260809 15:05:56.927484 25052 tablet_service.cc:1514] Processing DeleteTablet for tablet ad361cfefef24285887ae1fa010c41ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:56 UTC) from {username='slave'} at 127.0.0.1:52840
I20260809 15:05:56.927671 25248 tablet_service.cc:1514] Processing DeleteTablet for tablet ad361cfefef24285887ae1fa010c41ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:05:56 UTC) from {username='slave'} at 127.0.0.1:52554
I20260809 15:05:56.929214 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 24704
I20260809 15:05:56.938449 25399 tablet_replica.cc:330] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: stopping tablet replica
I20260809 15:05:56.938838 25399 raft_consensus.cc:2229] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:05:56.939292 25399 raft_consensus.cc:2258] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:56.943543 25399 ts_tablet_manager.cc:1905] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:56.949903 25403 tablet_replica.cc:330] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: stopping tablet replica
I20260809 15:05:56.950662 25399 ts_tablet_manager.cc:1918] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:56.950824 25399 log.cc:1196] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/ad361cfefef24285887ae1fa010c41ba
I20260809 15:05:56.951366 25399 ts_tablet_manager.cc:1939] T ad361cfefef24285887ae1fa010c41ba P a53a1319fca841d5ac72364860da4ea8: Deleting consensus metadata
I20260809 15:05:56.952163 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:56.930556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.27.186.129:32775)
0809 15:05:56.930732 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:56.930738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:56.943941 (+ 13203us) server_negotiation.cc:407] Connection header received
0809 15:05:56.944193 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:56.944198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:56.944293 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:56.944361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:56.944957 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.946097 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.947945 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:56.948614 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:56.948727 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:56.950391 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:56.950401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:56.950415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:56.951137 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:56.951194 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:56.951218 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:56.951307 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:56.951689 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 15:05:56.951807 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:05:56.953016 24582 catalog_manager.cc:4841] TS a53a1319fca841d5ac72364860da4ea8 (127.27.186.129:32775): tablet ad361cfefef24285887ae1fa010c41ba (table test-table [id=f5a9b17872c14201ab704223658a2b6e]) successfully deleted
I20260809 15:05:56.953362 24928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:56.957574 25403 raft_consensus.cc:2229] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:05:56.958244 25403 raft_consensus.cc:2258] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:56.958968 25402 tablet_replica.cc:330] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: stopping tablet replica
I20260809 15:05:56.959316 25402 raft_consensus.cc:2229] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:05:56.959738 25402 raft_consensus.cc:2258] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:05:56.961546 25403 ts_tablet_manager.cc:1905] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:56.963603 25402 ts_tablet_manager.cc:1905] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:05:56.969158 25403 ts_tablet_manager.cc:1918] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:56.969343 25403 log.cc:1196] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/ad361cfefef24285887ae1fa010c41ba
I20260809 15:05:56.970244 25403 ts_tablet_manager.cc:1939] T ad361cfefef24285887ae1fa010c41ba P 079c642f733040db97f65f7ec699dd3a: Deleting consensus metadata
I20260809 15:05:56.970517 25402 ts_tablet_manager.cc:1918] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:05:56.970711 25402 log.cc:1196] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/ad361cfefef24285887ae1fa010c41ba
I20260809 15:05:56.971285 25402 ts_tablet_manager.cc:1939] T ad361cfefef24285887ae1fa010c41ba P ffa9c1c644e141689911df35cdcb0e2c: Deleting consensus metadata
I20260809 15:05:56.971791 24584 catalog_manager.cc:4841] TS 079c642f733040db97f65f7ec699dd3a (127.27.186.131:34629): tablet ad361cfefef24285887ae1fa010c41ba (table test-table [id=f5a9b17872c14201ab704223658a2b6e]) successfully deleted
I20260809 15:05:56.972997 24584 catalog_manager.cc:4841] TS ffa9c1c644e141689911df35cdcb0e2c (127.27.186.130:44587): tablet ad361cfefef24285887ae1fa010c41ba (table test-table [id=f5a9b17872c14201ab704223658a2b6e]) successfully deleted
I20260809 15:05:57.044571 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 24704
I20260809 15:05:57.078770 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 24944
I20260809 15:05:57.086171 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:57.079662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.27.186.130:44587)
0809 15:05:57.079827 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:57.079832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:57.079851 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:05:57.079969 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:57.079973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:57.080054 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:57.080133 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:57.080573 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:57.081300 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:57.083220 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:57.083611 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:57.083701 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:57.084776 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:57.084786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:57.084799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:57.085340 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:57.085388 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:57.085407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:57.085493 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:57.085643 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 15:05:57.085744 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:05:57.086916 25135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:57.171965 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 24944
I20260809 15:05:57.204043 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 25149
I20260809 15:05:57.211840 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:57.204848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.27.186.131:34629)
0809 15:05:57.205031 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:57.205035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:57.205091 (+ 56us) server_negotiation.cc:407] Connection header received
0809 15:05:57.205289 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:57.205293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:57.205370 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:57.205451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:57.206068 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:57.206907 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:57.208784 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:57.209216 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:57.209312 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:57.210500 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:57.210508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:57.210522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:05:57.211103 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:57.211147 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:05:57.211167 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:57.211248 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:57.211418 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 15:05:57.211494 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 15:05:57.212510 25332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:57.327751 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 25149
I20260809 15:05:57.351428 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 24559
I20260809 15:05:57.352461 24657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:05:57.455297 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 24559
2026-08-09T15:05:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2706 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:05:57 dist-test-slave-q9h9 krb5kdc[25414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:05:57 dist-test-slave-q9h9 krb5kdc[25414](info): set up 2 sockets
Aug 09 15:05:57 dist-test-slave-q9h9 krb5kdc[25414](info): commencing operation
krb5kdc: starting...
W20260809 15:05:58.663480 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:05:58 dist-test-slave-q9h9 krb5kdc[25414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:05:58.888653 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:35521
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:59.197338 25439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:59.197597 25439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:59.197715 25439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:59.204598 25439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:05:59.204701 25439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:59.204779 25439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:59.204856 25439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:05:59.204916 25439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:05:59.214107 25439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:59.215201 25439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:59.216357 25439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:59.226878 25447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:59.226975 25446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:59.226948 25452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:59.227979 25439 server_base.cc:1034] running on GCE node
I20260809 15:05:59.228632 25439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:59.229790 25439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:59.230978 25439 hybrid_clock.cc:648] HybridClock initialized: now 1786287959230941 us; error 26 us; skew 500 ppm
I20260809 15:05:59.231262 25439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:59 dist-test-slave-q9h9 krb5kdc[25414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:59.234089 25439 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:05:59.235420 25439 webserver.cc:460] Webserver started at http://127.27.186.190:40265/ using document root <none> and password file <none>
I20260809 15:05:59.236068 25439 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:59.236202 25439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:59.236506 25439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:59.239208 25439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "28e9632df3b04855958050e72dd55416"
format_stamp: "Formatted at 2026-08-09 15:05:59 on dist-test-slave-q9h9"
server_key: "30898bfecab53c6dbd6b33d9059c7a15"
server_key_iv: "27dcfb395f29274322aa92d789a0cc93"
server_key_version: "encryptionkey@0"
I20260809 15:05:59.239900 25439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "28e9632df3b04855958050e72dd55416"
format_stamp: "Formatted at 2026-08-09 15:05:59 on dist-test-slave-q9h9"
server_key: "30898bfecab53c6dbd6b33d9059c7a15"
server_key_iv: "27dcfb395f29274322aa92d789a0cc93"
server_key_version: "encryptionkey@0"
I20260809 15:05:59.244620 25439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:05:59.247815 25464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:59.248967 25439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 15:05:59.249111 25439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "28e9632df3b04855958050e72dd55416"
format_stamp: "Formatted at 2026-08-09 15:05:59 on dist-test-slave-q9h9"
server_key: "30898bfecab53c6dbd6b33d9059c7a15"
server_key_iv: "27dcfb395f29274322aa92d789a0cc93"
server_key_version: "encryptionkey@0"
I20260809 15:05:59.249272 25439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:59.267233 25439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:59.271188 25439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:59.271440 25439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:59.287279 25439 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:35521
I20260809 15:05:59.287313 25542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:35521 every 8 connection(s)
I20260809 15:05:59.288470 25439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:05:59.291878 25543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:05:59.297511 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 25439
I20260809 15:05:59.297690 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:05:59.298267 28394 external_mini_cluster.cc:1428] Setting key 1aa3a1d4e09f1647974119f32fb6503f
I20260809 15:05:59.302250 25543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: Bootstrap starting.
I20260809 15:05:59.305532 25543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: Neither blocks nor log segments found. Creating new log.
I20260809 15:05:59.306598 25543 log.cc:824] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: Log is configured to *not* fsync() on all Append() calls
I20260809 15:05:59.309273 25543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: No bootstrap required, opened a new log
Aug 09 15:05:59 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:59.318583 25543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e9632df3b04855958050e72dd55416" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35521 } }
I20260809 15:05:59.319024 25543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:05:59.319137 25543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e9632df3b04855958050e72dd55416, State: Initialized, Role: FOLLOWER
I20260809 15:05:59.319746 25543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e9632df3b04855958050e72dd55416" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35521 } }
I20260809 15:05:59.320016 25543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:05:59.320159 25543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:05:59.320348 25543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:05:59.320992 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:59.299975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.27.186.190:35521)
0809 15:05:59.300689 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:59.300699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:59.300743 (+ 44us) server_negotiation.cc:407] Connection header received
0809 15:05:59.302400 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:59.302420 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:59.302748 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:59.303028 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:59.304799 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:59.306104 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:59.307709 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:59.308317 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:59.308436 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:59.313017 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:59.313027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:59.313043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:59.313081 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:59.316810 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:59.317431 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:59.317439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:59.317446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:59.317644 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:59.317974 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:59.317978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:59.317982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:59.318638 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:59.318926 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:59.319345 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 15:05:59.319716 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":343,"threads_started":1}
I20260809 15:05:59.322763 25543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e9632df3b04855958050e72dd55416" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35521 } }
I20260809 15:05:59.323203 25543 leader_election.cc:302] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e9632df3b04855958050e72dd55416; no voters:
I20260809 15:05:59.323751 25543 leader_election.cc:288] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:05:59.323841 25554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:05:59.324417 25554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [term 1 LEADER]: Becoming Leader. State: Replica: 28e9632df3b04855958050e72dd55416, State: Running, Role: LEADER
I20260809 15:05:59.324918 25554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e9632df3b04855958050e72dd55416" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35521 } }
I20260809 15:05:59.325676 25543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:05:59.327587 25557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28e9632df3b04855958050e72dd55416. Latest consensus state: current_term: 1 leader_uuid: "28e9632df3b04855958050e72dd55416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e9632df3b04855958050e72dd55416" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35521 } } }
I20260809 15:05:59.327661 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28e9632df3b04855958050e72dd55416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e9632df3b04855958050e72dd55416" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35521 } } }
I20260809 15:05:59.327951 25557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [sys.catalog]: This master's current role is: LEADER
I20260809 15:05:59.327982 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416 [sys.catalog]: This master's current role is: LEADER
W20260809 15:05:59.336674 25576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:05:59.336771 25576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:05:59.336879 25561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:05:59.338966 25561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:05:59.347018 25561 catalog_manager.cc:1347] Generated new cluster ID: 833c9b970ccf43549dc369d7fe1588c8
I20260809 15:05:59.347126 25561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:05:59.374964 25561 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:05:59.376711 25561 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:05:59.389413 25561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: Generated new TSK 0
I20260809 15:05:59.390148 25561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:05:59.455554 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:35521
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:05:59.781513 25583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:05:59.781775 25583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:05:59.781909 25583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:05:59.788908 25583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:05:59.789028 25583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:05:59.789191 25583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:05:59.798393 25583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:35521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:05:59.799496 25583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:05:59.800715 25583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:05:59.812764 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:59.812827 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:05:59.812827 25594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:05:59.813570 25583 server_base.cc:1034] running on GCE node
I20260809 15:05:59.814155 25583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:05:59.814996 25583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:05:59.816179 25583 hybrid_clock.cc:648] HybridClock initialized: now 1786287959816132 us; error 40 us; skew 500 ppm
I20260809 15:05:59.816458 25583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:05:59 dist-test-slave-q9h9 krb5kdc[25414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:05:59.819396 25583 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:05:59.820760 25583 webserver.cc:460] Webserver started at http://127.27.186.129:42633/ using document root <none> and password file <none>
I20260809 15:05:59.821400 25583 fs_manager.cc:359] Metadata directory not provided
I20260809 15:05:59.821502 25583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:05:59.821849 25583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:05:59.824813 25583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "34c1bdb080e44a32b185ae708c64fb82"
format_stamp: "Formatted at 2026-08-09 15:05:59 on dist-test-slave-q9h9"
server_key: "d5c0036e6f4f63f930d8269d05bef93a"
server_key_iv: "274c4a79beb5579b8c2faa9c0668f138"
server_key_version: "encryptionkey@0"
I20260809 15:05:59.825496 25583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "34c1bdb080e44a32b185ae708c64fb82"
format_stamp: "Formatted at 2026-08-09 15:05:59 on dist-test-slave-q9h9"
server_key: "d5c0036e6f4f63f930d8269d05bef93a"
server_key_iv: "274c4a79beb5579b8c2faa9c0668f138"
server_key_version: "encryptionkey@0"
I20260809 15:05:59.830314 25583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 15:05:59.833547 25608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:59.834761 25583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:05:59.834908 25583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "34c1bdb080e44a32b185ae708c64fb82"
format_stamp: "Formatted at 2026-08-09 15:05:59 on dist-test-slave-q9h9"
server_key: "d5c0036e6f4f63f930d8269d05bef93a"
server_key_iv: "274c4a79beb5579b8c2faa9c0668f138"
server_key_version: "encryptionkey@0"
I20260809 15:05:59.835062 25583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:05:59.852159 25583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:05:59.855851 25583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:05:59.856073 25583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:05:59.856917 25583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:05:59.858160 25583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:05:59.858232 25583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:59.858331 25583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:05:59.858374 25583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:05:59.888088 25583 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:34525
I20260809 15:05:59.888149 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:34525 every 8 connection(s)
I20260809 15:05:59.889371 25583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:05:59.898697 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 25583
I20260809 15:05:59.898860 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:05:59.899269 28394 external_mini_cluster.cc:1428] Setting key ffea2944456549d31af20cb72f94d310
Aug 09 15:05:59 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:05:59.907565 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:59.891639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:57867 (local address 127.27.186.190:35521)
0809 15:05:59.892055 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0809 15:05:59.892060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:05:59.892897 (+ 837us) server_negotiation.cc:407] Connection header received
0809 15:05:59.894726 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:05:59.894733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:05:59.894811 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:05:59.894920 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:05:59.896309 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:59.897030 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:59.898569 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:05:59.899113 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:05:59.899224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:59.902415 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:05:59.902421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:05:59.902426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:05:59.902455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:05:59.904759 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:59.905419 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:59.905423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:59.905425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:59.905538 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:05:59.905936 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:05:59.905940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:05:59.905942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:05:59.906230 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:05:59.906362 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 15:05:59.906997 (+ 635us) server_negotiation.cc:299] Negotiation successful
0809 15:05:59.907210 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":213,"threads_started":1}
I20260809 15:05:59.908362 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:05:59.892088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:35521 (local address 127.27.186.129:57867)
0809 15:05:59.892750 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 15:05:59.892792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 15:05:59.894467 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:05:59.895151 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:05:59.895159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:05:59.895634 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:05:59.896139 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:59.896155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:59.897193 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:59.897197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:59.898427 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:05:59.898439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:05:59.899348 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:05:59.899356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:05:59.899567 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:05:59.900150 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:05:59.900176 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:05:59.902209 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:05:59.904886 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:59.904895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:59.904908 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:59.905297 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:59.905640 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:05:59.905645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:05:59.905648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:05:59.905821 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:05:59.906349 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:05:59.906356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:05:59.906640 (+ 284us) client_negotiation.cc:769] Sending connection context
0809 15:05:59.906862 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 15:05:59.907327 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":253,"threads_started":1}
I20260809 15:05:59.909719 25773 heartbeater.cc:347] Connected to a master server at 127.27.186.190:35521
I20260809 15:05:59.909976 25773 heartbeater.cc:464] Registering TS with master...
I20260809 15:05:59.910650 25773 heartbeater.cc:511] Master 127.27.186.190:35521 requested a full tablet report, sending...
I20260809 15:05:59.912747 25493 ts_manager.cc:194] Registered new tserver with Master: 34c1bdb080e44a32b185ae708c64fb82 (127.27.186.129:34525)
I20260809 15:05:59.914127 25493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:57867
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:05:59.957273 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:35521
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:00.271473 25781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:00.271750 25781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:00.271888 25781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:00.278872 25781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:00.278977 25781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:00.279119 25781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:06:00.289361 25781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:35521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:00.290722 25781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:00.292183 25781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:00.305662 25790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:00.305750 25789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:00.306768 25792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:00.307062 25781 server_base.cc:1034] running on GCE node
I20260809 15:06:00.307586 25781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:00.308382 25781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:00.309559 25781 hybrid_clock.cc:648] HybridClock initialized: now 1786287960309505 us; error 48 us; skew 500 ppm
I20260809 15:06:00.309813 25781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:00 dist-test-slave-q9h9 krb5kdc[25414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:00.315421 25781 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:06:00.316730 25781 webserver.cc:460] Webserver started at http://127.27.186.130:39023/ using document root <none> and password file <none>
I20260809 15:06:00.317307 25781 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:00.317402 25781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:00.317646 25781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:00.320279 25781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "3d9958532937412a8ad52d5e8c10df61"
format_stamp: "Formatted at 2026-08-09 15:06:00 on dist-test-slave-q9h9"
server_key: "071daccceecfae4388e2aa570b3b7d79"
server_key_iv: "e002ada2cf4978a5dd422b6285991876"
server_key_version: "encryptionkey@0"
I20260809 15:06:00.320977 25781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "3d9958532937412a8ad52d5e8c10df61"
format_stamp: "Formatted at 2026-08-09 15:06:00 on dist-test-slave-q9h9"
server_key: "071daccceecfae4388e2aa570b3b7d79"
server_key_iv: "e002ada2cf4978a5dd422b6285991876"
server_key_version: "encryptionkey@0"
I20260809 15:06:00.325549 25781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260809 15:06:00.328773 25808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:00.329917 25781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:06:00.330094 25781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "3d9958532937412a8ad52d5e8c10df61"
format_stamp: "Formatted at 2026-08-09 15:06:00 on dist-test-slave-q9h9"
server_key: "071daccceecfae4388e2aa570b3b7d79"
server_key_iv: "e002ada2cf4978a5dd422b6285991876"
server_key_version: "encryptionkey@0"
I20260809 15:06:00.330252 25781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:00.340701 25781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:00.344069 25781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:00.344285 25781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:00.345052 25781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:00.346215 25781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:00.346289 25781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:00.346375 25781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:00.346419 25781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:00.376523 25781 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:46167
I20260809 15:06:00.376586 25991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:46167 every 8 connection(s)
I20260809 15:06:00.377645 25781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:06:00.378960 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 25781
I20260809 15:06:00.379184 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:06:00.379740 28394 external_mini_cluster.cc:1428] Setting key 2d3786e6c4e58469a2c8807d21115753
Aug 09 15:06:00 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:00.398319 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.380170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:59637 (local address 127.27.186.190:35521)
0809 15:06:00.380369 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.380375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.381664 (+ 1289us) server_negotiation.cc:407] Connection header received
0809 15:06:00.384021 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.384025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.384090 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.384168 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:00.385911 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.386654 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.388598 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.388996 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.389081 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.392585 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:00.392592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:00.392604 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:00.392639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:00.395476 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:00.396266 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:00.396269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:00.396271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:00.396381 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:00.396808 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:00.396810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:00.396812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:00.397064 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:00.397172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:00.397874 (+ 702us) server_negotiation.cc:299] Negotiation successful
0809 15:06:00.398084 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:06:00.399421 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.380708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:35521 (local address 127.27.186.130:59637)
0809 15:06:00.381488 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.381549 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.383744 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.384404 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.384415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.385063 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:00.385753 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.385776 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.386814 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.386820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.388478 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.388499 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.389194 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.389200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.389444 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.390091 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:00.390119 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:00.392369 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:00.395607 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:00.395616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:00.395630 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:00.396157 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:00.396485 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:00.396489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:00.396493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:00.396715 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:00.397182 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:00.397196 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:00.397533 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 15:06:00.397794 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 15:06:00.398255 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":289,"threads_started":1}
I20260809 15:06:00.400648 25993 heartbeater.cc:347] Connected to a master server at 127.27.186.190:35521
I20260809 15:06:00.400924 25993 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:00.401537 25993 heartbeater.cc:511] Master 127.27.186.190:35521 requested a full tablet report, sending...
I20260809 15:06:00.402880 25493 ts_manager.cc:194] Registered new tserver with Master: 3d9958532937412a8ad52d5e8c10df61 (127.27.186.130:46167)
I20260809 15:06:00.403659 25493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:59637
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:06:00.439186 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:35521
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:00.750164 26000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:00.750411 26000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:00.750545 26000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:00.757481 26000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:00.757591 26000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:00.757740 26000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:06:00.766872 26000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:35521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:00.767947 26000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:00.769119 26000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:00.781358 26009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:00.781374 26014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:00.781405 26012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:00.782290 26000 server_base.cc:1034] running on GCE node
I20260809 15:06:00.782814 26000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:00.783636 26000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:00.784814 26000 hybrid_clock.cc:648] HybridClock initialized: now 1786287960784766 us; error 44 us; skew 500 ppm
I20260809 15:06:00.785086 26000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:00 dist-test-slave-q9h9 krb5kdc[25414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:00.787935 26000 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:06:00.789261 26000 webserver.cc:460] Webserver started at http://127.27.186.131:41917/ using document root <none> and password file <none>
I20260809 15:06:00.789873 26000 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:00.789990 26000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:00.790297 26000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:00.792990 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "f9acfa5537f949ec80248914fffc67e4"
format_stamp: "Formatted at 2026-08-09 15:06:00 on dist-test-slave-q9h9"
server_key: "6ace010db56bcea138b4099b12dde698"
server_key_iv: "81bc7df1f5b2d3e6767c645973f9d9c3"
server_key_version: "encryptionkey@0"
I20260809 15:06:00.793699 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "f9acfa5537f949ec80248914fffc67e4"
format_stamp: "Formatted at 2026-08-09 15:06:00 on dist-test-slave-q9h9"
server_key: "6ace010db56bcea138b4099b12dde698"
server_key_iv: "81bc7df1f5b2d3e6767c645973f9d9c3"
server_key_version: "encryptionkey@0"
I20260809 15:06:00.798482 26000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 15:06:00.801596 26030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:00.802809 26000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:00.802950 26000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "f9acfa5537f949ec80248914fffc67e4"
format_stamp: "Formatted at 2026-08-09 15:06:00 on dist-test-slave-q9h9"
server_key: "6ace010db56bcea138b4099b12dde698"
server_key_iv: "81bc7df1f5b2d3e6767c645973f9d9c3"
server_key_version: "encryptionkey@0"
I20260809 15:06:00.803117 26000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:00.826153 26000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:00.830088 26000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:00.830353 26000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:00.831234 26000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:00.832470 26000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:00.832543 26000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:00.832653 26000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:00.832710 26000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:00.860348 26000 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:44727
I20260809 15:06:00.860405 26206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:44727 every 8 connection(s)
I20260809 15:06:00.861443 26000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:06:00.869756 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 26000
I20260809 15:06:00.869947 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:06:00.870538 28394 external_mini_cluster.cc:1428] Setting key 40e42b279f41e48b129e23b138f7ccb2
Aug 09 15:06:00 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:00.880021 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.863700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:42455 (local address 127.27.186.190:35521)
0809 15:06:00.863858 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.863863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.864929 (+ 1066us) server_negotiation.cc:407] Connection header received
0809 15:06:00.866731 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.866738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.866834 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.866924 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:00.868324 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.869075 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.871185 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.871707 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.871816 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.874995 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:00.874999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:00.875002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:00.875033 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:00.877256 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:00.877897 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:00.877901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:00.877904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:00.878060 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:00.878459 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:00.878462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:00.878464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:00.878720 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:00.878850 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:00.879474 (+ 624us) server_negotiation.cc:299] Negotiation successful
0809 15:06:00.879692 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:06:00.880769 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.864126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:35521 (local address 127.27.186.131:42455)
0809 15:06:00.864785 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.864820 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.866468 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.867163 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.867178 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.867644 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:00.868145 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.868162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.869226 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.869230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.871047 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.871060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.871916 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.871924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.872145 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.872705 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:00.872732 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:00.874798 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:00.877380 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:00.877388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:00.877401 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:00.877779 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:00.878157 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:00.878162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:00.878174 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:00.878344 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:00.878845 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:00.878852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:00.879119 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 15:06:00.879341 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 15:06:00.879776 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":262,"threads_started":1}
I20260809 15:06:00.881850 26207 heartbeater.cc:347] Connected to a master server at 127.27.186.190:35521
I20260809 15:06:00.882131 26207 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:00.882757 26207 heartbeater.cc:511] Master 127.27.186.190:35521 requested a full tablet report, sending...
I20260809 15:06:00.884116 25493 ts_manager.cc:194] Registered new tserver with Master: f9acfa5537f949ec80248914fffc67e4 (127.27.186.131:44727)
I20260809 15:06:00.885100 25493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:42455
I20260809 15:06:00.885360 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:06:00.901551 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.890114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.27.186.190:35521)
0809 15:06:00.890279 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.890284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.890697 (+ 413us) server_negotiation.cc:407] Connection header received
0809 15:06:00.890801 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.890806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.890869 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.890958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:00.892309 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.892957 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.894133 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.894513 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.894604 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.896964 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:00.896969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:00.896972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:00.896998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:00.899130 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:00.899866 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:00.899869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:00.899871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:00.899990 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:00.900360 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:00.900365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:00.900368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:00.900621 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:00.900825 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:00.901154 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 15:06:00.901299 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:06:00.904817 25493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:06:00.911218 25493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:06:00.916706 25773 heartbeater.cc:503] Master 127.27.186.190:35521 was elected leader, sending a full tablet report...
I20260809 15:06:00.932243 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.922180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.27.186.129:34525)
0809 15:06:00.923538 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.923548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.925051 (+ 1503us) server_negotiation.cc:407] Connection header received
0809 15:06:00.925270 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.925278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.925488 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.925652 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:00.926390 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.927835 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.929596 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.930952 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.931125 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.931289 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:00.931890 (+ 601us) server_negotiation.cc:299] Negotiation successful
0809 15:06:00.932018 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":1226,"threads_started":1}
I20260809 15:06:00.932984 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.923193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.27.186.130:46167)
0809 15:06:00.923969 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.923978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.925641 (+ 1663us) server_negotiation.cc:407] Connection header received
0809 15:06:00.925876 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.925883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.926137 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.926301 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:00.927119 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.928471 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.930104 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.931507 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.931676 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.931828 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:00.932460 (+ 632us) server_negotiation.cc:299] Negotiation successful
0809 15:06:00.932622 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":484,"threads_started":1}
I20260809 15:06:00.933117 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.922096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:34525 (local address 127.0.0.1:57706)
0809 15:06:00.924952 (+ 2856us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.924970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.925098 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.925745 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.925752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.925802 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:00.926241 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.926252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.927946 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.927951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.929449 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.929461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.931411 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.931418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.931599 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.931651 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 15:06:00.932462 (+ 811us) client_negotiation.cc:240] Negotiation successful
0809 15:06:00.932643 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2721,"mutex_wait_us":170,"spinlock_wait_cycles":4608,"thread_start_us":2421,"threads_started":1}
I20260809 15:06:00.933594 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.922496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:46167 (local address 127.0.0.1:33420)
0809 15:06:00.925526 (+ 3030us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.925543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.925661 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.926589 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.926596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.926619 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:00.926979 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.926987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.928608 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.928613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.929929 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.929941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.931966 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.931972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.932191 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.932225 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 15:06:00.932466 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 15:06:00.932670 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2917,"mutex_wait_us":49,"thread_start_us":2776,"threads_started":1}
I20260809 15:06:00.934803 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.922087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:44727 (local address 127.0.0.1:34554)
0809 15:06:00.924829 (+ 2742us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.924872 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.925035 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.925744 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.925751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.925791 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:00.926609 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.926619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.930119 (+ 3500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.930123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.932296 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.932309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.933995 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.934002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.934183 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.934215 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:06:00.934319 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 15:06:00.934439 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2141,"mutex_wait_us":304,"spinlock_wait_cycles":2304,"thread_start_us":341,"threads_started":1}
I20260809 15:06:00.935762 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.922750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.27.186.131:44727)
0809 15:06:00.924489 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.924497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.924975 (+ 478us) server_negotiation.cc:407] Connection header received
0809 15:06:00.925254 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.925263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.925494 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.926118 (+ 624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:00.926843 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.929995 (+ 3152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.932450 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.933803 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.933982 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:00.934171 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:00.935317 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0809 15:06:00.935477 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":1416,"threads_started":1}
I20260809 15:06:00.936923 25884 tablet_service.cc:1467] Processing CreateTablet for tablet b6501dcd7b6748f0b852090f2af34aff (DEFAULT_TABLE table=test-table [id=f37b4d045715417ca6466ee175ccd0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:00.936920 25679 tablet_service.cc:1467] Processing CreateTablet for tablet b6501dcd7b6748f0b852090f2af34aff (DEFAULT_TABLE table=test-table [id=f37b4d045715417ca6466ee175ccd0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:00.938010 25884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6501dcd7b6748f0b852090f2af34aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:00.938055 25679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6501dcd7b6748f0b852090f2af34aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:00.941370 26106 tablet_service.cc:1467] Processing CreateTablet for tablet b6501dcd7b6748f0b852090f2af34aff (DEFAULT_TABLE table=test-table [id=f37b4d045715417ca6466ee175ccd0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:00.942704 26106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6501dcd7b6748f0b852090f2af34aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:00.949182 26228 tablet_bootstrap.cc:492] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Bootstrap starting.
I20260809 15:06:00.951362 26229 tablet_bootstrap.cc:492] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Bootstrap starting.
I20260809 15:06:00.952566 26228 tablet_bootstrap.cc:654] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:00.953816 26228 log.cc:824] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:00.954291 26229 tablet_bootstrap.cc:654] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:00.955495 26229 log.cc:824] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:00.956835 26228 tablet_bootstrap.cc:492] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: No bootstrap required, opened a new log
I20260809 15:06:00.957140 26228 ts_tablet_manager.cc:1397] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 15:06:00.957666 26234 tablet_bootstrap.cc:492] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Bootstrap starting.
I20260809 15:06:00.960501 26229 tablet_bootstrap.cc:492] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: No bootstrap required, opened a new log
I20260809 15:06:00.960772 26234 tablet_bootstrap.cc:654] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:00.960779 26229 ts_tablet_manager.cc:1397] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 15:06:00.961941 26234 log.cc:824] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:00.964774 26228 raft_consensus.cc:348] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.965167 26228 raft_consensus.cc:374] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:00.965265 26228 raft_consensus.cc:729] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9958532937412a8ad52d5e8c10df61, State: Initialized, Role: FOLLOWER
I20260809 15:06:00.965427 26234 tablet_bootstrap.cc:492] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: No bootstrap required, opened a new log
I20260809 15:06:00.965751 26234 ts_tablet_manager.cc:1397] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260809 15:06:00.965885 26228 consensus_queue.cc:260] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.966917 25993 heartbeater.cc:503] Master 127.27.186.190:35521 was elected leader, sending a full tablet report...
I20260809 15:06:00.967382 26228 ts_tablet_manager.cc:1428] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260809 15:06:00.970839 26229 raft_consensus.cc:348] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.971125 26229 raft_consensus.cc:374] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:00.971220 26229 raft_consensus.cc:729] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c1bdb080e44a32b185ae708c64fb82, State: Initialized, Role: FOLLOWER
I20260809 15:06:00.971735 26229 consensus_queue.cc:260] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.973311 26229 ts_tablet_manager.cc:1428] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260809 15:06:00.973645 26234 raft_consensus.cc:348] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.973951 26234 raft_consensus.cc:374] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:00.974064 26234 raft_consensus.cc:729] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9acfa5537f949ec80248914fffc67e4, State: Initialized, Role: FOLLOWER
I20260809 15:06:00.974506 26234 consensus_queue.cc:260] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.975420 26207 heartbeater.cc:503] Master 127.27.186.190:35521 was elected leader, sending a full tablet report...
I20260809 15:06:00.976743 26234 ts_tablet_manager.cc:1428] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260809 15:06:00.993393 26239 raft_consensus.cc:482] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:06:00.993610 26239 raft_consensus.cc:504] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:00.995061 26239 leader_election.cc:288] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9acfa5537f949ec80248914fffc67e4 (127.27.186.131:44727), 3d9958532937412a8ad52d5e8c10df61 (127.27.186.130:46167)
I20260809 15:06:01.001377 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.995394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:37075 (local address 127.27.186.131:44727)
0809 15:06:00.995561 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.995567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.996198 (+ 631us) server_negotiation.cc:407] Connection header received
0809 15:06:00.996368 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.996374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.996442 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.996520 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:00.997114 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.997844 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.999757 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.000509 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.000653 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.000719 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:01.001029 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 15:06:01.001141 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:06:01.001379 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.995501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:50275 (local address 127.27.186.130:46167)
0809 15:06:00.995681 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:00.995688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:00.996352 (+ 664us) server_negotiation.cc:407] Connection header received
0809 15:06:00.996484 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:00.996489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:00.996556 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:00.996647 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:00.997212 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.997877 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.999747 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.000479 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.000600 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.000669 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:01.001018 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 15:06:01.001148 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 15:06:01.001498 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.995390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:44727 (local address 127.27.186.129:37075)
0809 15:06:00.996086 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.996103 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.996242 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.996544 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.996549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.996572 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:00.996973 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.996983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.997973 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.997978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.999606 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.999619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.000742 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.000747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:01.000879 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.000902 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:06:01.000985 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 15:06:01.001103 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":78,"thread_start_us":354,"threads_started":1}
I20260809 15:06:01.001549 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:00.995400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:46167 (local address 127.27.186.129:50275)
0809 15:06:00.996245 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:00.996259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:00.996379 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:00.996674 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:00.996681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:00.996709 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:00.997062 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.997073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:00.998009 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:00.998040 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:00.999584 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:00.999604 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.000697 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.000703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:01.000854 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.000886 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 15:06:01.000957 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 15:06:01.001073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":28,"thread_start_us":549,"threads_started":1}
I20260809 15:06:01.002352 26131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6501dcd7b6748f0b852090f2af34aff" candidate_uuid: "34c1bdb080e44a32b185ae708c64fb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9acfa5537f949ec80248914fffc67e4" is_pre_election: true
I20260809 15:06:01.002379 25927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6501dcd7b6748f0b852090f2af34aff" candidate_uuid: "34c1bdb080e44a32b185ae708c64fb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9958532937412a8ad52d5e8c10df61" is_pre_election: true
I20260809 15:06:01.002736 26131 raft_consensus.cc:2454] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34c1bdb080e44a32b185ae708c64fb82 in term 0.
I20260809 15:06:01.002767 25927 raft_consensus.cc:2454] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34c1bdb080e44a32b185ae708c64fb82 in term 0.
I20260809 15:06:01.003314 25615 leader_election.cc:302] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c1bdb080e44a32b185ae708c64fb82, 3d9958532937412a8ad52d5e8c10df61; no voters:
I20260809 15:06:01.003654 26239 raft_consensus.cc:2781] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:06:01.003808 26239 raft_consensus.cc:482] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:06:01.003907 26239 raft_consensus.cc:3037] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:01.005954 26239 raft_consensus.cc:504] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:01.006544 26239 leader_election.cc:288] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [CANDIDATE]: Term 1 election: Requested vote from peers f9acfa5537f949ec80248914fffc67e4 (127.27.186.131:44727), 3d9958532937412a8ad52d5e8c10df61 (127.27.186.130:46167)
I20260809 15:06:01.006916 26131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6501dcd7b6748f0b852090f2af34aff" candidate_uuid: "34c1bdb080e44a32b185ae708c64fb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9acfa5537f949ec80248914fffc67e4"
I20260809 15:06:01.006949 25927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6501dcd7b6748f0b852090f2af34aff" candidate_uuid: "34c1bdb080e44a32b185ae708c64fb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9958532937412a8ad52d5e8c10df61"
I20260809 15:06:01.007073 26131 raft_consensus.cc:3037] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:01.007148 25927 raft_consensus.cc:3037] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:01.009186 26131 raft_consensus.cc:2454] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34c1bdb080e44a32b185ae708c64fb82 in term 1.
I20260809 15:06:01.009255 25927 raft_consensus.cc:2454] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34c1bdb080e44a32b185ae708c64fb82 in term 1.
I20260809 15:06:01.009564 25610 leader_election.cc:302] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c1bdb080e44a32b185ae708c64fb82, f9acfa5537f949ec80248914fffc67e4; no voters:
I20260809 15:06:01.009806 26239 raft_consensus.cc:2781] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:01.010218 26239 raft_consensus.cc:686] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 1 LEADER]: Becoming Leader. State: Replica: 34c1bdb080e44a32b185ae708c64fb82, State: Running, Role: LEADER
I20260809 15:06:01.010600 26239 consensus_queue.cc:237] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } }
I20260809 15:06:01.014601 25491 catalog_manager.cc:5495] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34c1bdb080e44a32b185ae708c64fb82 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "34c1bdb080e44a32b185ae708c64fb82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c1bdb080e44a32b185ae708c64fb82" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 } health_report { overall_health: UNKNOWN } } }
I20260809 15:06:01.034524 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:01.029184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.27.186.129:34525)
0809 15:06:01.029358 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:01.029365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:01.029486 (+ 121us) server_negotiation.cc:407] Connection header received
0809 15:06:01.029664 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:01.029667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:01.029748 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:01.029873 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:01.030309 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.031036 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.032469 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.032854 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.032957 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.033530 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:01.033975 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:01.034144 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:01.034286 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 15:06:01.034328 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:06:01.041314 25927 raft_consensus.cc:1261] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 1 FOLLOWER]: Refusing update from remote peer 34c1bdb080e44a32b185ae708c64fb82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:01.041330 26131 raft_consensus.cc:1261] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 1 FOLLOWER]: Refusing update from remote peer 34c1bdb080e44a32b185ae708c64fb82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:01.042145 26243 consensus_queue.cc:1035] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9958532937412a8ad52d5e8c10df61" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 46167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:01.042440 26239 consensus_queue.cc:1035] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9acfa5537f949ec80248914fffc67e4" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:01.053702 26249 mvcc.cc:205] Tried to move back new op lower bound from 7316635488417665024 to 7316635488302465024. Current Snapshot: MvccSnapshot[applied={T|T < 7316635488417665024}]
I20260809 15:06:01.064433 25491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:01.064694 25491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:01.068161 25491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28e9632df3b04855958050e72dd55416: Sending DeleteTablet for 3 replicas of tablet b6501dcd7b6748f0b852090f2af34aff
I20260809 15:06:01.069077 25679 tablet_service.cc:1514] Processing DeleteTablet for tablet b6501dcd7b6748f0b852090f2af34aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:01 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:57706
I20260809 15:06:01.069100 26106 tablet_service.cc:1514] Processing DeleteTablet for tablet b6501dcd7b6748f0b852090f2af34aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:01 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:34554
I20260809 15:06:01.069238 25884 tablet_service.cc:1514] Processing DeleteTablet for tablet b6501dcd7b6748f0b852090f2af34aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:01 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:33420
I20260809 15:06:01.071421 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 25583
I20260809 15:06:01.073082 26264 tablet_replica.cc:330] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: stopping tablet replica
I20260809 15:06:01.073439 26264 raft_consensus.cc:2229] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:01.073875 26264 raft_consensus.cc:2258] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:01.076021 26264 ts_tablet_manager.cc:1905] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:01.082969 26264 ts_tablet_manager.cc:1918] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:01.083158 26264 log.cc:1196] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/b6501dcd7b6748f0b852090f2af34aff
I20260809 15:06:01.083714 26264 ts_tablet_manager.cc:1939] T b6501dcd7b6748f0b852090f2af34aff P f9acfa5537f949ec80248914fffc67e4: Deleting consensus metadata
I20260809 15:06:01.085469 25467 catalog_manager.cc:4841] TS f9acfa5537f949ec80248914fffc67e4 (127.27.186.131:44727): tablet b6501dcd7b6748f0b852090f2af34aff (table test-table [id=f37b4d045715417ca6466ee175ccd0ec]) successfully deleted
I20260809 15:06:01.089084 26265 tablet_replica.cc:330] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: stopping tablet replica
I20260809 15:06:01.089591 26265 raft_consensus.cc:2229] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:06:01.090304 26265 raft_consensus.cc:2258] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:01.092432 26265 ts_tablet_manager.cc:1905] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 15:06:01 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:06:01.093019 26266 tablet_replica.cc:330] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: stopping tablet replica
I20260809 15:06:01.093389 26266 raft_consensus.cc:2229] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:01.093865 26266 raft_consensus.cc:2258] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:01.098264 26266 ts_tablet_manager.cc:1905] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:01.100693 26265 ts_tablet_manager.cc:1918] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:01.100957 26265 log.cc:1196] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/b6501dcd7b6748f0b852090f2af34aff
I20260809 15:06:01.101477 26265 ts_tablet_manager.cc:1939] T b6501dcd7b6748f0b852090f2af34aff P 34c1bdb080e44a32b185ae708c64fb82: Deleting consensus metadata
I20260809 15:06:01.103171 25468 catalog_manager.cc:4841] TS 34c1bdb080e44a32b185ae708c64fb82 (127.27.186.129:34525): tablet b6501dcd7b6748f0b852090f2af34aff (table test-table [id=f37b4d045715417ca6466ee175ccd0ec]) successfully deleted
I20260809 15:06:01.103890 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:01.073516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.27.186.129:34525)
0809 15:06:01.073677 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:01.073683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:01.082334 (+ 8651us) server_negotiation.cc:407] Connection header received
0809 15:06:01.082713 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:01.082718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:01.082792 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:01.082875 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:01.084785 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.085923 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.087878 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.088625 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.088727 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.094161 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:01.094175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:01.094191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:01.094230 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:01.098437 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:01.099311 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:01.099320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:01.099324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:01.099514 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:01.101863 (+ 2349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:01.101868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:01.101872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:01.102357 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:01.102685 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:01.103275 (+ 590us) server_negotiation.cc:299] Negotiation successful
0809 15:06:01.103526 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:06:01.104943 26266 ts_tablet_manager.cc:1918] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:01.105073 26266 log.cc:1196] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/b6501dcd7b6748f0b852090f2af34aff
I20260809 15:06:01.105314 25767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:01.105609 26266 ts_tablet_manager.cc:1939] T b6501dcd7b6748f0b852090f2af34aff P 3d9958532937412a8ad52d5e8c10df61: Deleting consensus metadata
I20260809 15:06:01.107321 25469 catalog_manager.cc:4841] TS 3d9958532937412a8ad52d5e8c10df61 (127.27.186.130:46167): tablet b6501dcd7b6748f0b852090f2af34aff (table test-table [id=f37b4d045715417ca6466ee175ccd0ec]) successfully deleted
I20260809 15:06:01.200239 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 25583
I20260809 15:06:01.228477 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 25781
Aug 09 15:06:01 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:06:01.242213 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:01.229263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.27.186.130:46167)
0809 15:06:01.229434 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:01.229439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:01.229458 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:06:01.229530 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:01.229534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:01.229595 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:01.229679 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:01.230988 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.231818 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.232983 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.233392 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.233511 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.237482 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:01.237495 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:01.237509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:01.237541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:01.240010 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:01.240602 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:01.240609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:01.240611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:01.240726 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:01.240993 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:01.240996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:01.240998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:01.241348 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:01.241563 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:01.241764 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 15:06:01.241916 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:06:01.242913 25983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:01.336545 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 25781
I20260809 15:06:01.359714 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 26000
Aug 09 15:06:01 dist-test-slave-q9h9 krb5kdc[25414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:06:01.374392 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:01.360432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.27.186.131:44727)
0809 15:06:01.360613 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:01.360618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:01.360652 (+ 34us) server_negotiation.cc:407] Connection header received
0809 15:06:01.360821 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:01.360825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:01.360902 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:01.360982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:01.362305 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.363029 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.364297 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:01.364663 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:01.364744 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:01.368859 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:01.368876 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:01.368898 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:01.368938 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:01.371767 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:01.372392 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:01.372400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:01.372402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:01.372528 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:01.372881 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:01.372885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:01.372888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:01.373238 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:01.373528 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:01.373822 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 15:06:01.374050 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:06:01.375021 26196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:01.460578 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 26000
I20260809 15:06:01.480471 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 25439
I20260809 15:06:01.481468 25535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:01.586815 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 25439
2026-08-09T15:06:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T15:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:01Z Disabled control of system clock
I20260809 15:06:01.672399 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:41677
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 15:06:01.986304 26280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:01.986557 26280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:01.986634 26280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:01.993551 26280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:01.993647 26280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:01.993685 26280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:01.993739 26280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:01.993788 26280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:02.002452 26280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:41677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:02.003508 26280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:02.005128 26280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:02.015893 26291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:02.016062 26294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:02.016425 26289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:02.016511 26280 server_base.cc:1034] running on GCE node
I20260809 15:06:02.017241 26280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:02.018502 26280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:02.019707 26280 hybrid_clock.cc:648] HybridClock initialized: now 1786287962019664 us; error 31 us; skew 500 ppm
I20260809 15:06:02.020015 26280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:02.022035 26280 webserver.cc:460] Webserver started at http://127.27.186.190:35615/ using document root <none> and password file <none>
I20260809 15:06:02.022652 26280 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:02.022761 26280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:02.023051 26280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:02.025679 26280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "9cf477475e4d437b9a4469f43d5d48e6"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "bbb6fc38cb614e0dbf3e39f3aeef7609"
server_key_iv: "b9dac97fba9ac85adc0820671bfff2cd"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.026444 26280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "9cf477475e4d437b9a4469f43d5d48e6"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "bbb6fc38cb614e0dbf3e39f3aeef7609"
server_key_iv: "b9dac97fba9ac85adc0820671bfff2cd"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.031005 26280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 15:06:02.034171 26303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:02.035359 26280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:06:02.035516 26280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "9cf477475e4d437b9a4469f43d5d48e6"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "bbb6fc38cb614e0dbf3e39f3aeef7609"
server_key_iv: "b9dac97fba9ac85adc0820671bfff2cd"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.035705 26280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:02.062887 26280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:02.063593 26280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:02.063853 26280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:02.080173 26280 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:41677
I20260809 15:06:02.080204 26380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:41677 every 8 connection(s)
I20260809 15:06:02.081393 26280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:02.082748 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 26280
I20260809 15:06:02.082904 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:02.083325 28394 external_mini_cluster.cc:1428] Setting key 919cd612e14b6427951413d984c55c23
I20260809 15:06:02.085517 26382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:02.098658 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:02.085063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.27.186.190:41677)
0809 15:06:02.086190 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:02.086205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:02.086257 (+ 52us) server_negotiation.cc:407] Connection header received
0809 15:06:02.088508 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:02.088537 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:02.089122 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:02.089554 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:02.090113 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:02.092122 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:02.093480 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:02.094549 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:02.094737 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:02.095836 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:02.095847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:02.095868 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:02.096612 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:02.096704 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:02.096736 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:02.096858 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:02.097048 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 15:06:02.097376 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":362,"threads_started":1}
I20260809 15:06:02.100037 26382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6: Bootstrap starting.
I20260809 15:06:02.103044 26382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:02.104048 26382 log.cc:824] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:02.106793 26382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6: No bootstrap required, opened a new log
I20260809 15:06:02.113966 26382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf477475e4d437b9a4469f43d5d48e6" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41677 } }
I20260809 15:06:02.114339 26382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:02.114444 26382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf477475e4d437b9a4469f43d5d48e6, State: Initialized, Role: FOLLOWER
I20260809 15:06:02.115054 26382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf477475e4d437b9a4469f43d5d48e6" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41677 } }
I20260809 15:06:02.115309 26382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:02.115499 26382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:02.115693 26382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:02.118045 26382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf477475e4d437b9a4469f43d5d48e6" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41677 } }
I20260809 15:06:02.118506 26382 leader_election.cc:302] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf477475e4d437b9a4469f43d5d48e6; no voters:
I20260809 15:06:02.119010 26382 leader_election.cc:288] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:02.119145 26393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:02.119866 26393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf477475e4d437b9a4469f43d5d48e6, State: Running, Role: LEADER
I20260809 15:06:02.120342 26393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf477475e4d437b9a4469f43d5d48e6" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41677 } }
I20260809 15:06:02.120913 26382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:02.123154 26399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf477475e4d437b9a4469f43d5d48e6. Latest consensus state: current_term: 1 leader_uuid: "9cf477475e4d437b9a4469f43d5d48e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf477475e4d437b9a4469f43d5d48e6" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41677 } } }
I20260809 15:06:02.123620 26399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:02.123385 26396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cf477475e4d437b9a4469f43d5d48e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf477475e4d437b9a4469f43d5d48e6" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 41677 } } }
I20260809 15:06:02.123736 26396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:02.125484 26407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:02.129993 26407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:02.138219 26407 catalog_manager.cc:1347] Generated new cluster ID: 8cae2d2b3b9c4889ba74bf6f8b19edfa
I20260809 15:06:02.138330 26407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:02.156121 26407 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:02.157838 26407 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:02.164726 26407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6: Generated new TSK 0
I20260809 15:06:02.165594 26407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:02.170828 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41677
--builtin_ntp_servers=127.27.186.148:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 15:06:02.503033 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:02.503278 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:02.503399 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:02.510277 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:02.510357 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:02.510489 26423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:06:02.519022 26423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:02.520018 26423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:02.521637 26423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:02.533869 26431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:02.534045 26433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:02.534421 26423 server_base.cc:1034] running on GCE node
W20260809 15:06:02.534492 26436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:02.535075 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:02.535923 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:02.537134 26423 hybrid_clock.cc:648] HybridClock initialized: now 1786287962537088 us; error 37 us; skew 500 ppm
I20260809 15:06:02.537444 26423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:02.539542 26423 webserver.cc:460] Webserver started at http://127.27.186.129:41859/ using document root <none> and password file <none>
I20260809 15:06:02.540177 26423 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:02.540277 26423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:02.540556 26423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:02.543227 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "2fb2adc1324a4a6db766edcce9d471b8"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "3d92768fbfd6a5f5c5e5c595037b6775"
server_key_iv: "d4608b52433f98540dd8e07bb9e55b1f"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.543946 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "2fb2adc1324a4a6db766edcce9d471b8"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "3d92768fbfd6a5f5c5e5c595037b6775"
server_key_iv: "d4608b52433f98540dd8e07bb9e55b1f"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.548589 26423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:06:02.551805 26446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:02.552927 26423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:06:02.553073 26423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "2fb2adc1324a4a6db766edcce9d471b8"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "3d92768fbfd6a5f5c5e5c595037b6775"
server_key_iv: "d4608b52433f98540dd8e07bb9e55b1f"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.553265 26423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:02.569568 26423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:02.570283 26423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:02.570550 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:02.571403 26423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:02.572695 26423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:02.572793 26423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:02.572894 26423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:02.572973 26423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:02.602816 26423 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:37371
I20260809 15:06:02.602877 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:37371 every 8 connection(s)
I20260809 15:06:02.604000 26423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:06:02.609771 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 26423
I20260809 15:06:02.609930 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:06:02.610374 28394 external_mini_cluster.cc:1428] Setting key 17b85ca595fc8fdfefcfefbf29514d5f
I20260809 15:06:02.613251 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41677
--builtin_ntp_servers=127.27.186.148:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 15:06:02.616173 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:02.606192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:45815 (local address 127.27.186.190:41677)
0809 15:06:02.606599 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:02.606603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:02.607590 (+ 987us) server_negotiation.cc:407] Connection header received
0809 15:06:02.609278 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:02.609284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:02.609360 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:02.609453 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:02.610574 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:02.611417 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:02.613561 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:02.613965 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:02.614102 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:02.615497 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:02.615501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:02.615505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:02.615520 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:02.615565 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:02.615577 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:02.615644 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:02.615849 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 15:06:02.615921 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":216,"threads_started":1}
I20260809 15:06:02.617274 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:02.606620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41677 (local address 127.27.186.129:45815)
0809 15:06:02.607443 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:02.607483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:02.609025 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:02.609695 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:02.609703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:02.609782 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:02.610376 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:02.610394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:02.611592 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:02.611598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:02.613373 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:02.613389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:02.614208 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:02.614215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:02.614479 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:02.615171 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:06:02.615185 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:02.615233 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:06:02.615323 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:02.615664 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:02.615675 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:02.615682 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:06:02.615841 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 15:06:02.616164 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":248,"threads_started":1}
I20260809 15:06:02.618738 26616 heartbeater.cc:347] Connected to a master server at 127.27.186.190:41677
I20260809 15:06:02.619004 26616 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:02.619673 26616 heartbeater.cc:511] Master 127.27.186.190:41677 requested a full tablet report, sending...
I20260809 15:06:02.621718 26331 ts_manager.cc:194] Registered new tserver with Master: 2fb2adc1324a4a6db766edcce9d471b8 (127.27.186.129:37371)
I20260809 15:06:02.623607 26331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.129:45815
W20260809 15:06:02.933338 26622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:02.933588 26622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:02.933719 26622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:02.940670 26622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:02.940778 26622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:02.940913 26622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:06:02.949431 26622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:02.950546 26622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:02.952204 26622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:02.964403 26629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:02.964469 26628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:02.964443 26636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:02.965188 26622 server_base.cc:1034] running on GCE node
I20260809 15:06:02.965708 26622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:02.966572 26622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:02.967763 26622 hybrid_clock.cc:648] HybridClock initialized: now 1786287962967722 us; error 30 us; skew 500 ppm
I20260809 15:06:02.968039 26622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:02.970005 26622 webserver.cc:460] Webserver started at http://127.27.186.130:43673/ using document root <none> and password file <none>
I20260809 15:06:02.970674 26622 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:02.970767 26622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:02.971050 26622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:02.973589 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "ca9c50ddb0e24f80ba88879f5c83dd26"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "b993c74303fd696fd147f0950c4777e6"
server_key_iv: "14afd849278c32be2b509377dc618c60"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.974367 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "ca9c50ddb0e24f80ba88879f5c83dd26"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "b993c74303fd696fd147f0950c4777e6"
server_key_iv: "14afd849278c32be2b509377dc618c60"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.978963 26622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:06:02.982085 26646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:02.983235 26622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:06:02.983379 26622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "ca9c50ddb0e24f80ba88879f5c83dd26"
format_stamp: "Formatted at 2026-08-09 15:06:02 on dist-test-slave-q9h9"
server_key: "b993c74303fd696fd147f0950c4777e6"
server_key_iv: "14afd849278c32be2b509377dc618c60"
server_key_version: "encryptionkey@0"
I20260809 15:06:02.983561 26622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:03.001339 26622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:03.001993 26622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:03.002290 26622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:03.003110 26622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:03.004390 26622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:03.004482 26622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:03.004585 26622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:03.004653 26622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:03.033519 26622 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:34179
I20260809 15:06:03.033572 26807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:34179 every 8 connection(s)
I20260809 15:06:03.034626 26622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:06:03.041625 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 26622
I20260809 15:06:03.041812 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:06:03.042387 28394 external_mini_cluster.cc:1428] Setting key 93b9ed6929d74345fb6ddabf266d5dcc
I20260809 15:06:03.045298 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41677
--builtin_ntp_servers=127.27.186.148:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 15:06:03.046273 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.036815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:60959 (local address 127.27.186.190:41677)
0809 15:06:03.036982 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.036986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.038176 (+ 1190us) server_negotiation.cc:407] Connection header received
0809 15:06:03.039832 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.039836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.039902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.039984 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:03.040894 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.041586 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.043518 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.044133 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.044261 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.045603 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:03.045607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:03.045611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:03.045627 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:03.045659 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:03.045671 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:03.045745 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.045939 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.046010 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:06:03.047438 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.037202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41677 (local address 127.27.186.130:60959)
0809 15:06:03.038006 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.038073 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.039617 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.040186 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.040193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.040271 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:03.040759 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.040774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.041740 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.041746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.043377 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.043390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.044342 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.044350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.044618 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.045274 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:06:03.045278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:03.045326 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:06:03.045424 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:03.045758 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:03.045768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:03.045775 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.045935 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.046321 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":258,"threads_started":1}
I20260809 15:06:03.048611 26811 heartbeater.cc:347] Connected to a master server at 127.27.186.190:41677
I20260809 15:06:03.048869 26811 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:03.049521 26811 heartbeater.cc:511] Master 127.27.186.190:41677 requested a full tablet report, sending...
I20260809 15:06:03.050781 26331 ts_manager.cc:194] Registered new tserver with Master: ca9c50ddb0e24f80ba88879f5c83dd26 (127.27.186.130:34179)
I20260809 15:06:03.051535 26331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.130:60959
W20260809 15:06:03.366092 26815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:03.366405 26815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:03.366526 26815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:03.373749 26815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:03.373852 26815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:03.373993 26815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:06:03.383188 26815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:03.384410 26815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:03.386246 26815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:03.399255 26823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:03.399312 26827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:03.399343 26825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:03.399998 26815 server_base.cc:1034] running on GCE node
I20260809 15:06:03.400525 26815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:03.401371 26815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:03.402578 26815 hybrid_clock.cc:648] HybridClock initialized: now 1786287963402536 us; error 33 us; skew 500 ppm
I20260809 15:06:03.402877 26815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:03.404969 26815 webserver.cc:460] Webserver started at http://127.27.186.131:38771/ using document root <none> and password file <none>
I20260809 15:06:03.405606 26815 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:03.405712 26815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:03.406000 26815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:03.408748 26815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "33180fdd1c8b4c75bec13a14391b67b5"
format_stamp: "Formatted at 2026-08-09 15:06:03 on dist-test-slave-q9h9"
server_key: "1b3ae792b3bd8cd16b99c35919105b93"
server_key_iv: "047ab3170c40d6ef86546b107a06d969"
server_key_version: "encryptionkey@0"
I20260809 15:06:03.409497 26815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "33180fdd1c8b4c75bec13a14391b67b5"
format_stamp: "Formatted at 2026-08-09 15:06:03 on dist-test-slave-q9h9"
server_key: "1b3ae792b3bd8cd16b99c35919105b93"
server_key_iv: "047ab3170c40d6ef86546b107a06d969"
server_key_version: "encryptionkey@0"
I20260809 15:06:03.414407 26815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260809 15:06:03.417577 26835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:03.418723 26815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:03.418875 26815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "33180fdd1c8b4c75bec13a14391b67b5"
format_stamp: "Formatted at 2026-08-09 15:06:03 on dist-test-slave-q9h9"
server_key: "1b3ae792b3bd8cd16b99c35919105b93"
server_key_iv: "047ab3170c40d6ef86546b107a06d969"
server_key_version: "encryptionkey@0"
I20260809 15:06:03.419055 26815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:03.455080 26815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:03.455775 26815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:03.456054 26815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:03.456919 26815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:03.458248 26815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:03.458346 26815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:03.458451 26815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:03.458520 26815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:03.487807 26815 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:35757
I20260809 15:06:03.487861 27008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:35757 every 8 connection(s)
I20260809 15:06:03.488953 26815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:06:03.492851 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 26815
I20260809 15:06:03.493032 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:06:03.493598 28394 external_mini_cluster.cc:1428] Setting key 3110cdb89997a6fb41b3e973333a71b9
I20260809 15:06:03.500555 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.491168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:41245 (local address 127.27.186.190:41677)
0809 15:06:03.491325 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.491329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.492629 (+ 1300us) server_negotiation.cc:407] Connection header received
0809 15:06:03.494893 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.494898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.494970 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.495048 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:03.496164 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.496933 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.498413 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.498804 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.498896 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.499971 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:03.499976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:03.499979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:03.499995 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:03.500026 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:03.500038 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:03.500104 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.500250 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.500321 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:06:03.501365 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.491617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:41677 (local address 127.27.186.131:41245)
0809 15:06:03.492444 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.492505 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.494610 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.495277 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.495286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.495401 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:03.495975 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.495992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.497077 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.497081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.498297 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.498310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.498965 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.498970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.499151 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.499692 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:06:03.499694 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:03.499728 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:06:03.499810 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:03.500107 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:03.500112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:03.500117 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.500234 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.500478 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":263,"threads_started":1}
I20260809 15:06:03.502473 27009 heartbeater.cc:347] Connected to a master server at 127.27.186.190:41677
I20260809 15:06:03.502724 27009 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:03.503378 27009 heartbeater.cc:511] Master 127.27.186.190:41677 requested a full tablet report, sending...
I20260809 15:06:03.504616 26331 ts_manager.cc:194] Registered new tserver with Master: 33180fdd1c8b4c75bec13a14391b67b5 (127.27.186.131:35757)
I20260809 15:06:03.505362 26331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.131:41245
I20260809 15:06:03.508262 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:06:03.517578 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.511965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.27.186.190:41677)
0809 15:06:03.512134 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.512139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.512541 (+ 402us) server_negotiation.cc:407] Connection header received
0809 15:06:03.512692 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.512697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.512764 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.512843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:03.513321 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.513977 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.515395 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.515774 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.515888 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.516897 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:03.516902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:03.516905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:03.516920 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:03.516949 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:03.516959 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:03.517076 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.517271 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.517324 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 15:06:03.520802 26331 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:06:03.527468 26331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:06:03.559600 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.27.186.130:34179)
0809 15:06:03.550116 (+ 11582us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.550125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.550886 (+ 761us) server_negotiation.cc:407] Connection header received
0809 15:06:03.550994 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.551002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.551240 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.552256 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:03.552460 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.554069 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.556924 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.558437 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.558583 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.558808 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.559088 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.559253 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11400,"thread_start_us":11303,"threads_started":1}
I20260809 15:06:03.560014 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.538383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34179 (local address 127.0.0.1:40542)
0809 15:06:03.550495 (+ 12112us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.550528 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.550705 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.551477 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.551485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.551527 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:03.552008 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.552023 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.554230 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.554236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.556703 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.556717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.558597 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.558603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.558872 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.558912 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.559026 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.559310 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11900,"mutex_wait_us":205,"thread_start_us":11757,"threads_started":1}
I20260809 15:06:03.560804 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.538765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:37371 (local address 127.0.0.1:55656)
0809 15:06:03.552322 (+ 13557us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.552339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.552543 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.553561 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.553566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.553593 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:03.553969 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.553980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.555864 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.555869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.558261 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.558275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.560097 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.560104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.560283 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.560321 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.560405 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.560525 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13433,"mutex_wait_us":4,"thread_start_us":13268,"threads_started":1}
I20260809 15:06:03.561614 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.539096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.27.186.129:37371)
0809 15:06:03.551694 (+ 12598us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.551702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.552435 (+ 733us) server_negotiation.cc:407] Connection header received
0809 15:06:03.552893 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.552902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.553128 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.553283 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:03.554190 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.555694 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.558449 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.559822 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.560056 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.560251 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.561132 (+ 881us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.561288 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12429,"thread_start_us":12596,"threads_started":1}
I20260809 15:06:03.563387 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.552347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:35757 (local address 127.0.0.1:48760)
0809 15:06:03.555941 (+ 3594us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.555958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.556061 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.556940 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.556944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.556968 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:03.557349 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.557362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.558758 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.558763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.560771 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.560786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.562641 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.562647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.562809 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.562840 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.562936 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.563051 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3473,"mutex_wait_us":35,"thread_start_us":3876,"threads_started":1}
I20260809 15:06:03.563519 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.551221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.27.186.131:35757)
0809 15:06:03.556408 (+ 5187us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.556416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.556449 (+ 33us) server_negotiation.cc:407] Connection header received
0809 15:06:03.556545 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.556552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.556758 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.556935 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:03.557525 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.558596 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.560982 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.562120 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.562317 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.562521 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.563015 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.563170 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4335,"thread_start_us":4192,"threads_started":1}
I20260809 15:06:03.564178 26717 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 (DEFAULT_TABLE table=test-table [id=4b72972a09cc4d639d2c5b8b1da071c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:03.565317 26717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:03.565282 26525 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 (DEFAULT_TABLE table=test-table [id=4b72972a09cc4d639d2c5b8b1da071c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:03.566926 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:03.567147 26905 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 (DEFAULT_TABLE table=test-table [id=4b72972a09cc4d639d2c5b8b1da071c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:03.568514 26905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:03.576157 27033 tablet_bootstrap.cc:492] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Bootstrap starting.
I20260809 15:06:03.579334 27033 tablet_bootstrap.cc:654] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:03.580307 27033 log.cc:824] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:03.582788 27039 tablet_bootstrap.cc:492] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Bootstrap starting.
I20260809 15:06:03.586404 27039 tablet_bootstrap.cc:654] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:03.587033 27037 tablet_bootstrap.cc:492] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Bootstrap starting.
I20260809 15:06:03.587704 27039 log.cc:824] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:03.588729 27033 tablet_bootstrap.cc:492] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: No bootstrap required, opened a new log
I20260809 15:06:03.589042 27033 ts_tablet_manager.cc:1397] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260809 15:06:03.590471 27037 tablet_bootstrap.cc:654] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:03.591531 27037 log.cc:824] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:03.592432 27039 tablet_bootstrap.cc:492] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: No bootstrap required, opened a new log
I20260809 15:06:03.592651 27039 ts_tablet_manager.cc:1397] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260809 15:06:03.594353 27037 tablet_bootstrap.cc:492] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: No bootstrap required, opened a new log
I20260809 15:06:03.594605 27037 ts_tablet_manager.cc:1397] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:06:03.596529 27033 raft_consensus.cc:348] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.596840 27033 raft_consensus.cc:374] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:03.596935 27033 raft_consensus.cc:729] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9c50ddb0e24f80ba88879f5c83dd26, State: Initialized, Role: FOLLOWER
I20260809 15:06:03.597451 27033 consensus_queue.cc:260] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.598470 26811 heartbeater.cc:503] Master 127.27.186.190:41677 was elected leader, sending a full tablet report...
I20260809 15:06:03.599082 27033 ts_tablet_manager.cc:1428] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260809 15:06:03.602906 27039 raft_consensus.cc:348] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.603353 27039 raft_consensus.cc:374] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:03.603487 27039 raft_consensus.cc:729] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33180fdd1c8b4c75bec13a14391b67b5, State: Initialized, Role: FOLLOWER
I20260809 15:06:03.604193 27039 consensus_queue.cc:260] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.604574 27037 raft_consensus.cc:348] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.605059 27037 raft_consensus.cc:374] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:03.605170 27037 raft_consensus.cc:729] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb2adc1324a4a6db766edcce9d471b8, State: Initialized, Role: FOLLOWER
I20260809 15:06:03.605228 27009 heartbeater.cc:503] Master 127.27.186.190:41677 was elected leader, sending a full tablet report...
I20260809 15:06:03.605734 27037 consensus_queue.cc:260] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.607137 27039 ts_tablet_manager.cc:1428] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260809 15:06:03.610110 26616 heartbeater.cc:503] Master 127.27.186.190:41677 was elected leader, sending a full tablet report...
I20260809 15:06:03.610754 27037 ts_tablet_manager.cc:1428] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260809 15:06:03.740451 27011 tablet.cc:2369] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:03.745177 27046 raft_consensus.cc:482] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:06:03.745437 27046 raft_consensus.cc:504] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.746987 27046 leader_election.cc:288] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9c50ddb0e24f80ba88879f5c83dd26 (127.27.186.130:34179), 33180fdd1c8b4c75bec13a14391b67b5 (127.27.186.131:35757)
I20260809 15:06:03.753607 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.747327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:34179 (local address 127.27.186.129:41059)
0809 15:06:03.747966 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.747983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.748113 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.748525 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.748530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.748558 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:03.748889 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.748902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.750076 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.750082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.751693 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.751705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.752951 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.752958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.753120 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.753153 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.753237 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.753364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":333,"threads_started":1}
I20260809 15:06:03.753644 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.747386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:41059 (local address 127.27.186.130:34179)
0809 15:06:03.747540 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.747544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.748060 (+ 516us) server_negotiation.cc:407] Connection header received
0809 15:06:03.748252 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.748257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.748356 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.748444 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:03.749037 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.749917 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.751883 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.752761 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.752869 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.752916 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.753285 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.753410 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:06:03.754002 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.748245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:33025 (local address 127.27.186.131:35757)
0809 15:06:03.748433 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.748438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.748733 (+ 295us) server_negotiation.cc:407] Connection header received
0809 15:06:03.748866 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.748872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.748942 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.749027 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:03.749638 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.750551 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.752288 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.753135 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.753297 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.753362 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.753666 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.753795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:06:03.754042 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.748056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:35757 (local address 127.27.186.129:33025)
0809 15:06:03.748628 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:03.748642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:03.748754 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:03.749056 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:03.749060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:03.749084 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:03.749468 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.749479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.750670 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.750673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.752092 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:03.752106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.753323 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.753330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:03.753496 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.753531 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 15:06:03.753628 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 15:06:03.753743 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":301,"threads_started":1}
I20260809 15:06:03.754693 26746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf3bf7a9d84ed2a3516edd9cc165f5" candidate_uuid: "2fb2adc1324a4a6db766edcce9d471b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" is_pre_election: true
I20260809 15:06:03.755009 26931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf3bf7a9d84ed2a3516edd9cc165f5" candidate_uuid: "2fb2adc1324a4a6db766edcce9d471b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33180fdd1c8b4c75bec13a14391b67b5" is_pre_election: true
I20260809 15:06:03.755270 26746 raft_consensus.cc:2454] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb2adc1324a4a6db766edcce9d471b8 in term 0.
I20260809 15:06:03.755517 26931 raft_consensus.cc:2454] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb2adc1324a4a6db766edcce9d471b8 in term 0.
I20260809 15:06:03.755964 26451 leader_election.cc:302] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb2adc1324a4a6db766edcce9d471b8, ca9c50ddb0e24f80ba88879f5c83dd26; no voters:
I20260809 15:06:03.756299 27046 raft_consensus.cc:2781] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:06:03.756461 27046 raft_consensus.cc:482] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:06:03.756569 27046 raft_consensus.cc:3037] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:03.758728 27046 raft_consensus.cc:504] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.759253 27046 leader_election.cc:288] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [CANDIDATE]: Term 1 election: Requested vote from peers ca9c50ddb0e24f80ba88879f5c83dd26 (127.27.186.130:34179), 33180fdd1c8b4c75bec13a14391b67b5 (127.27.186.131:35757)
I20260809 15:06:03.759613 26746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf3bf7a9d84ed2a3516edd9cc165f5" candidate_uuid: "2fb2adc1324a4a6db766edcce9d471b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26"
I20260809 15:06:03.759733 26931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf3bf7a9d84ed2a3516edd9cc165f5" candidate_uuid: "2fb2adc1324a4a6db766edcce9d471b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33180fdd1c8b4c75bec13a14391b67b5"
I20260809 15:06:03.759819 26746 raft_consensus.cc:3037] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:03.759914 26931 raft_consensus.cc:3037] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:03.762933 26931 raft_consensus.cc:2454] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb2adc1324a4a6db766edcce9d471b8 in term 1.
I20260809 15:06:03.762939 26746 raft_consensus.cc:2454] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb2adc1324a4a6db766edcce9d471b8 in term 1.
I20260809 15:06:03.763288 26451 leader_election.cc:302] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb2adc1324a4a6db766edcce9d471b8, ca9c50ddb0e24f80ba88879f5c83dd26; no voters:
I20260809 15:06:03.763530 27046 raft_consensus.cc:2781] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:03.763919 27046 raft_consensus.cc:686] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2fb2adc1324a4a6db766edcce9d471b8, State: Running, Role: LEADER
I20260809 15:06:03.764325 27046 consensus_queue.cc:237] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } }
I20260809 15:06:03.768143 26329 catalog_manager.cc:5495] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb2adc1324a4a6db766edcce9d471b8 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "2fb2adc1324a4a6db766edcce9d471b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb2adc1324a4a6db766edcce9d471b8" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 37371 } health_report { overall_health: HEALTHY } } }
I20260809 15:06:03.782220 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.777122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.27.186.129:37371)
0809 15:06:03.777293 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.777299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.777441 (+ 142us) server_negotiation.cc:407] Connection header received
0809 15:06:03.777623 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.777626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.777702 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.777819 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:03.778226 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.778873 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.780206 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.780597 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.780704 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.781241 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:03.781657 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:03.781803 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.781954 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.782007 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 15:06:03.786412 26812 tablet.cc:2369] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:03.789603 26931 raft_consensus.cc:1261] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 1 FOLLOWER]: Refusing update from remote peer 2fb2adc1324a4a6db766edcce9d471b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:03.789603 26746 raft_consensus.cc:1261] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 1 FOLLOWER]: Refusing update from remote peer 2fb2adc1324a4a6db766edcce9d471b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:03.790421 27052 consensus_queue.cc:1035] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9c50ddb0e24f80ba88879f5c83dd26" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:03.790729 27046 consensus_queue.cc:1035] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33180fdd1c8b4c75bec13a14391b67b5" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:03.800597 27056 mvcc.cc:205] Tried to move back new op lower bound from 7316635499673616384 to 7316635499581800448. Current Snapshot: MvccSnapshot[applied={T|T < 7316635499673616384}]
I20260809 15:06:03.802977 27058 mvcc.cc:205] Tried to move back new op lower bound from 7316635499673616384 to 7316635499581800448. Current Snapshot: MvccSnapshot[applied={T|T < 7316635499673616384}]
I20260809 15:06:03.813884 26331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:03.814126 26331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:03.817219 26331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cf477475e4d437b9a4469f43d5d48e6: Sending DeleteTablet for 3 replicas of tablet a3bf3bf7a9d84ed2a3516edd9cc165f5
I20260809 15:06:03.818264 26717 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:03 UTC) from {username='slave'} at 127.0.0.1:40542
I20260809 15:06:03.818311 26905 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:03 UTC) from {username='slave'} at 127.0.0.1:48760
I20260809 15:06:03.819017 26525 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:03 UTC) from {username='slave'} at 127.0.0.1:55656
I20260809 15:06:03.819209 27071 tablet_replica.cc:330] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: stopping tablet replica
I20260809 15:06:03.819581 27071 raft_consensus.cc:2229] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:03.819820 27073 tablet_replica.cc:330] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: stopping tablet replica
I20260809 15:06:03.820052 27071 raft_consensus.cc:2258] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:03.820309 27073 raft_consensus.cc:2229] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:06:03.820763 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 26423
I20260809 15:06:03.822576 27073 raft_consensus.cc:2258] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:03.826053 27073 ts_tablet_manager.cc:1905] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:03.829746 27071 ts_tablet_manager.cc:1905] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:03.833458 27073 ts_tablet_manager.cc:1918] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:03.835932 27072 tablet_replica.cc:330] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: stopping tablet replica
I20260809 15:06:03.836403 27072 raft_consensus.cc:2229] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:03.836895 27072 raft_consensus.cc:2258] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:03.837352 27073 log.cc:1196] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/a3bf3bf7a9d84ed2a3516edd9cc165f5
I20260809 15:06:03.837461 27071 ts_tablet_manager.cc:1918] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:03.837702 27071 log.cc:1196] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/a3bf3bf7a9d84ed2a3516edd9cc165f5
I20260809 15:06:03.838048 27073 ts_tablet_manager.cc:1939] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 2fb2adc1324a4a6db766edcce9d471b8: Deleting consensus metadata
I20260809 15:06:03.838336 27071 ts_tablet_manager.cc:1939] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P 33180fdd1c8b4c75bec13a14391b67b5: Deleting consensus metadata
I20260809 15:06:03.839887 26307 catalog_manager.cc:4841] TS 33180fdd1c8b4c75bec13a14391b67b5 (127.27.186.131:35757): tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 (table test-table [id=4b72972a09cc4d639d2c5b8b1da071c6]) successfully deleted
I20260809 15:06:03.840374 27072 ts_tablet_manager.cc:1905] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:03.840962 26304 catalog_manager.cc:4841] TS 2fb2adc1324a4a6db766edcce9d471b8 (127.27.186.129:37371): tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 (table test-table [id=4b72972a09cc4d639d2c5b8b1da071c6]) successfully deleted
I20260809 15:06:03.843104 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.821570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.27.186.129:37371)
0809 15:06:03.822182 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.822187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.833962 (+ 11775us) server_negotiation.cc:407] Connection header received
0809 15:06:03.834263 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.834269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.834355 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.834434 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:03.835064 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.836216 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.838310 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.838967 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.839062 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.841537 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:03.841547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:03.841562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:03.842293 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:03.842351 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:03.842375 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:03.842480 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.842640 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.842750 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260809 15:06:03.843720 26610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:03.846442 27072 ts_tablet_manager.cc:1918] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:03.846639 27072 log.cc:1196] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/a3bf3bf7a9d84ed2a3516edd9cc165f5
I20260809 15:06:03.847193 27072 ts_tablet_manager.cc:1939] T a3bf3bf7a9d84ed2a3516edd9cc165f5 P ca9c50ddb0e24f80ba88879f5c83dd26: Deleting consensus metadata
I20260809 15:06:03.848685 26307 catalog_manager.cc:4841] TS ca9c50ddb0e24f80ba88879f5c83dd26 (127.27.186.130:34179): tablet a3bf3bf7a9d84ed2a3516edd9cc165f5 (table test-table [id=4b72972a09cc4d639d2c5b8b1da071c6]) successfully deleted
I20260809 15:06:03.937655 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 26423
I20260809 15:06:03.960332 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 26622
I20260809 15:06:03.967005 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:03.961143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.27.186.130:34179)
0809 15:06:03.961343 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:03.961349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:03.961370 (+ 21us) server_negotiation.cc:407] Connection header received
0809 15:06:03.961526 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:03.961530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:03.961625 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:03.961724 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:03.962170 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.962943 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.964169 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:03.964576 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:03.964686 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:03.965659 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:03.965669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:03.965682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:03.966287 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:03.966333 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:03.966354 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:03.966471 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:03.966584 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 15:06:03.966686 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:06:03.967654 26800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:04.053197 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 26622
I20260809 15:06:04.077870 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 26815
I20260809 15:06:04.085247 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:04.078612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.27.186.131:35757)
0809 15:06:04.078768 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:04.078779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:04.078799 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:06:04.078959 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:04.078963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:04.079049 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:04.079140 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:04.079484 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:04.080566 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:04.081814 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:04.082652 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:04.082771 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:04.083727 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:04.083738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:04.083757 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:04.084471 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:04.084530 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:04.084555 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:04.084677 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:04.084822 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 15:06:04.084926 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:06:04.085904 27000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:04.171408 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 26815
I20260809 15:06:04.195773 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 26280
I20260809 15:06:04.196982 26374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:04.312016 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 26280
2026-08-09T15:06:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2726 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:04 dist-test-slave-q9h9 krb5kdc[27088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:04 dist-test-slave-q9h9 krb5kdc[27088](info): set up 2 sockets
Aug 09 15:06:04 dist-test-slave-q9h9 krb5kdc[27088](info): commencing operation
krb5kdc: starting...
W20260809 15:06:05.518216 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:05 dist-test-slave-q9h9 krb5kdc[27088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:05.747253 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:45561
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:06.050513 27103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:06.050770 27103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:06.050905 27103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:06.057821 27103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:06.057930 27103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:06.058001 27103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:06.058099 27103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:06.058163 27103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:06.067229 27103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:45561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:45561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:06.068308 27103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:06.069494 27103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:06.080258 27115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:06.080296 27120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:06.080330 27114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:06.081395 27103 server_base.cc:1034] running on GCE node
I20260809 15:06:06.082103 27103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:06.083321 27103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:06.084529 27103 hybrid_clock.cc:648] HybridClock initialized: now 1786287966084475 us; error 42 us; skew 500 ppm
I20260809 15:06:06.084842 27103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:06 dist-test-slave-q9h9 krb5kdc[27088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:06.093039 27103 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:06.094472 27103 webserver.cc:460] Webserver started at http://127.27.186.190:43313/ using document root <none> and password file <none>
I20260809 15:06:06.095114 27103 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:06.095232 27103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:06.095504 27103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:06.098372 27103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "e005c3ec276249409afb0e1f25f50a0d"
format_stamp: "Formatted at 2026-08-09 15:06:06 on dist-test-slave-q9h9"
server_key: "3fdd0a3f162683b47cb4dced4c27883f"
server_key_iv: "5a7787e5a14261d91bf1fe3f7e1d20c9"
server_key_version: "encryptionkey@0"
I20260809 15:06:06.099076 27103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "e005c3ec276249409afb0e1f25f50a0d"
format_stamp: "Formatted at 2026-08-09 15:06:06 on dist-test-slave-q9h9"
server_key: "3fdd0a3f162683b47cb4dced4c27883f"
server_key_iv: "5a7787e5a14261d91bf1fe3f7e1d20c9"
server_key_version: "encryptionkey@0"
I20260809 15:06:06.103924 27103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 15:06:06.107165 27131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:06.108340 27103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260809 15:06:06.108495 27103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "e005c3ec276249409afb0e1f25f50a0d"
format_stamp: "Formatted at 2026-08-09 15:06:06 on dist-test-slave-q9h9"
server_key: "3fdd0a3f162683b47cb4dced4c27883f"
server_key_iv: "5a7787e5a14261d91bf1fe3f7e1d20c9"
server_key_version: "encryptionkey@0"
I20260809 15:06:06.108664 27103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:06.131326 27103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:06.135201 27103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:06.135447 27103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:06.152660 27103 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:45561
I20260809 15:06:06.152679 27202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:45561 every 8 connection(s)
I20260809 15:06:06.153929 27103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:06.156257 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 27103
I20260809 15:06:06.156446 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:06.157070 28394 external_mini_cluster.cc:1428] Setting key 15f720153c0ca99e569ef6c7660da215
I20260809 15:06:06.157851 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:06.169132 27204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d: Bootstrap starting.
Aug 09 15:06:06 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:06.172150 27204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:06.173174 27204 log.cc:824] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:06.175724 27204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d: No bootstrap required, opened a new log
I20260809 15:06:06.177763 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:06.158955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.27.186.190:45561)
0809 15:06:06.159685 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:06.159696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:06.159733 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:06:06.161267 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:06.161288 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:06.161641 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:06.162005 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:06.163458 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:06.164738 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:06.165992 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:06.166722 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:06.166926 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:06.170761 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:06.170772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:06.170787 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:06.170831 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:06.173694 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:06.174376 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:06.174382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:06.174391 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:06.174553 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:06.174942 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:06.174946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:06.174948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:06.175415 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:06.175682 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:06.176547 (+ 865us) server_negotiation.cc:299] Negotiation successful
0809 15:06:06.176841 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":329,"threads_started":1}
I20260809 15:06:06.183310 27204 raft_consensus.cc:348] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005c3ec276249409afb0e1f25f50a0d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45561 } }
I20260809 15:06:06.183627 27204 raft_consensus.cc:374] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:06.183712 27204 raft_consensus.cc:729] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e005c3ec276249409afb0e1f25f50a0d, State: Initialized, Role: FOLLOWER
I20260809 15:06:06.184257 27204 consensus_queue.cc:260] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005c3ec276249409afb0e1f25f50a0d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45561 } }
I20260809 15:06:06.184491 27204 raft_consensus.cc:388] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:06.184617 27204 raft_consensus.cc:482] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:06.184763 27204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:06.186784 27204 raft_consensus.cc:504] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005c3ec276249409afb0e1f25f50a0d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45561 } }
I20260809 15:06:06.187202 27204 leader_election.cc:302] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e005c3ec276249409afb0e1f25f50a0d; no voters:
I20260809 15:06:06.187744 27204 leader_election.cc:288] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:06.187865 27212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:06.188462 27212 raft_consensus.cc:686] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [term 1 LEADER]: Becoming Leader. State: Replica: e005c3ec276249409afb0e1f25f50a0d, State: Running, Role: LEADER
I20260809 15:06:06.189015 27212 consensus_queue.cc:237] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005c3ec276249409afb0e1f25f50a0d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45561 } }
I20260809 15:06:06.189517 27204 sys_catalog.cc:564] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:06.191915 27214 sys_catalog.cc:455] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e005c3ec276249409afb0e1f25f50a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005c3ec276249409afb0e1f25f50a0d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45561 } } }
I20260809 15:06:06.191954 27216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e005c3ec276249409afb0e1f25f50a0d. Latest consensus state: current_term: 1 leader_uuid: "e005c3ec276249409afb0e1f25f50a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005c3ec276249409afb0e1f25f50a0d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45561 } } }
I20260809 15:06:06.192301 27214 sys_catalog.cc:458] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:06.192301 27216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:06.193764 27221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:06.200973 27221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:06.209017 27221 catalog_manager.cc:1347] Generated new cluster ID: 6b68679bd33340cd80d1c79c35f3aa3b
I20260809 15:06:06.209122 27221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:06.222051 27221 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:06.223755 27221 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:06.230799 27221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d: Generated new TSK 0
I20260809 15:06:06.231669 27221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.186.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
Entry for principal kudu/127.27.186.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab.
I20260809 15:06:06.307211 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45561
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:06.628615 27243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:06.628844 27243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:06.628971 27243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:06.635934 27243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:06.636052 27243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:06.636211 27243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:06:06.645219 27243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.129.keytab
--principal=kudu/127.27.186.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:06.646267 27243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:06.647374 27243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:06.659309 27250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:06.659344 27249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:06.659956 27254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:06.660179 27243 server_base.cc:1034] running on GCE node
I20260809 15:06:06.660740 27243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:06.661568 27243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:06.662765 27243 hybrid_clock.cc:648] HybridClock initialized: now 1786287966662706 us; error 44 us; skew 500 ppm
I20260809 15:06:06.663045 27243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:06 dist-test-slave-q9h9 krb5kdc[27088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:06.665894 27243 init.cc:377] Logged in from keytab as kudu/127.27.186.129@KRBTEST.COM (short username kudu)
I20260809 15:06:06.667183 27243 webserver.cc:460] Webserver started at http://127.27.186.129:37707/ using document root <none> and password file <none>
I20260809 15:06:06.667791 27243 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:06.667909 27243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:06.668203 27243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:06.670933 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "1100fac73d3e4b498d023ae887415016"
format_stamp: "Formatted at 2026-08-09 15:06:06 on dist-test-slave-q9h9"
server_key: "c86b7445442c8aece09cbf02ce951824"
server_key_iv: "fa08f01db92a32e5231797ba99546270"
server_key_version: "encryptionkey@0"
I20260809 15:06:06.671639 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "1100fac73d3e4b498d023ae887415016"
format_stamp: "Formatted at 2026-08-09 15:06:06 on dist-test-slave-q9h9"
server_key: "c86b7445442c8aece09cbf02ce951824"
server_key_iv: "fa08f01db92a32e5231797ba99546270"
server_key_version: "encryptionkey@0"
I20260809 15:06:06.676369 27243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:06:06.680246 27267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:06.681392 27243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 15:06:06.681536 27243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "1100fac73d3e4b498d023ae887415016"
format_stamp: "Formatted at 2026-08-09 15:06:06 on dist-test-slave-q9h9"
server_key: "c86b7445442c8aece09cbf02ce951824"
server_key_iv: "fa08f01db92a32e5231797ba99546270"
server_key_version: "encryptionkey@0"
I20260809 15:06:06.681700 27243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:06.697544 27243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:06.700908 27243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:06.701146 27243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:06.701947 27243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:06.703151 27243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:06.703223 27243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:06.703310 27243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:06.703352 27243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:06.733621 27243 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:43901
I20260809 15:06:06.733681 27444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:43901 every 8 connection(s)
I20260809 15:06:06.734803 27243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:06:06.741226 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 27243
I20260809 15:06:06.741412 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:06:06.741957 28394 external_mini_cluster.cc:1428] Setting key e2415e6f6e06a0c6cab69528e4bf320e
Aug 09 15:06:06 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.129@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:06.754103 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:06.737078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:48155 (local address 127.27.186.190:45561)
0809 15:06:06.737504 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:06.737508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:06.738320 (+ 812us) server_negotiation.cc:407] Connection header received
0809 15:06:06.740042 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:06.740047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:06.740122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:06.740226 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:06.741934 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:06.742691 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:06.744558 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:06.745208 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:06.745321 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:06.749003 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:06.749007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:06.749011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:06.749042 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:06.751338 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:06.752100 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:06.752103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:06.752105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:06.752213 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:06.752631 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:06.752633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:06.752635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:06.752874 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:06.752986 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:06.753665 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 15:06:06.753827 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":240,"threads_started":1}
I20260809 15:06:06.755235 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:06.737479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:45561 (local address 127.27.186.129:48155)
0809 15:06:06.738183 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:06.738220 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:06.739795 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:06.740447 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:06.740455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:06.741020 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:06.741761 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:06.741788 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:06.742874 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:06.742878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:06.744404 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:06.744419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:06.745428 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:06.745435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:06.745708 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:06.746370 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:06.746398 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:06.748803 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:06.751472 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:06.751480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:06.751493 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:06.752006 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:06.752304 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:06.752308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:06.752311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:06.752539 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:06.752985 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:06.752992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:06.753337 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 15:06:06.753607 (+ 270us) client_negotiation.cc:240] Negotiation successful
0809 15:06:06.754072 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260809 15:06:06.756614 27446 heartbeater.cc:347] Connected to a master server at 127.27.186.190:45561
I20260809 15:06:06.756875 27446 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:06.757647 27446 heartbeater.cc:511] Master 127.27.186.190:45561 requested a full tablet report, sending...
I20260809 15:06:06.759692 27156 ts_manager.cc:194] Registered new tserver with Master: 1100fac73d3e4b498d023ae887415016 (127.27.186.129:43901)
I20260809 15:06:06.761075 27156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.129@KRBTEST.COM'} at 127.27.186.129:48155
WARNING: no policy specified for kudu/127.27.186.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
Entry for principal kudu/127.27.186.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab.
I20260809 15:06:06.799204 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45561
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:07.108455 27461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:07.108709 27461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:07.108841 27461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:07.115839 27461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:07.115933 27461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:07.116106 27461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:06:07.125252 27461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.130.keytab
--principal=kudu/127.27.186.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:07.126298 27461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:07.127414 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:07.139230 27470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:07.139231 27471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:07.139266 27474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:07.140551 27461 server_base.cc:1034] running on GCE node
I20260809 15:06:07.141122 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:07.141938 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:07.143170 27461 hybrid_clock.cc:648] HybridClock initialized: now 1786287967143107 us; error 56 us; skew 500 ppm
I20260809 15:06:07.143460 27461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:07 dist-test-slave-q9h9 krb5kdc[27088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:07.146294 27461 init.cc:377] Logged in from keytab as kudu/127.27.186.130@KRBTEST.COM (short username kudu)
I20260809 15:06:07.147567 27461 webserver.cc:460] Webserver started at http://127.27.186.130:44321/ using document root <none> and password file <none>
I20260809 15:06:07.148172 27461 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:07.148288 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:07.148568 27461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:07.151206 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "0c1d046ff0ec4d3488035cee97c0fe2f"
format_stamp: "Formatted at 2026-08-09 15:06:07 on dist-test-slave-q9h9"
server_key: "1d70675ac6b08293b106063cec8e8e14"
server_key_iv: "1d915a40003e2105adc7059d3476a482"
server_key_version: "encryptionkey@0"
I20260809 15:06:07.151899 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "0c1d046ff0ec4d3488035cee97c0fe2f"
format_stamp: "Formatted at 2026-08-09 15:06:07 on dist-test-slave-q9h9"
server_key: "1d70675ac6b08293b106063cec8e8e14"
server_key_iv: "1d915a40003e2105adc7059d3476a482"
server_key_version: "encryptionkey@0"
I20260809 15:06:07.156884 27461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 15:06:07.160190 27492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:07.161419 27461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 15:06:07.161563 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "0c1d046ff0ec4d3488035cee97c0fe2f"
format_stamp: "Formatted at 2026-08-09 15:06:07 on dist-test-slave-q9h9"
server_key: "1d70675ac6b08293b106063cec8e8e14"
server_key_iv: "1d915a40003e2105adc7059d3476a482"
server_key_version: "encryptionkey@0"
I20260809 15:06:07.161729 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:07.176172 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:07.179587 27461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:07.179836 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:07.180658 27461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:07.181809 27461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:07.181885 27461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:07.182006 27461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:07.182090 27461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 15:06:07.212348 27461 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:38851
I20260809 15:06:07.212420 27657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:38851 every 8 connection(s)
I20260809 15:06:07.213558 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:06:07.219123 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 27461
I20260809 15:06:07.219271 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:06:07.219687 28394 external_mini_cluster.cc:1428] Setting key 375a4d70ec9aa8b99b2c2c16c6a4a43e
Aug 09 15:06:07 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.130@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:07.232456 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.215753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:56843 (local address 127.27.186.190:45561)
0809 15:06:07.215948 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.215955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.217041 (+ 1086us) server_negotiation.cc:407] Connection header received
0809 15:06:07.218734 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.218738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.218806 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.218900 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:07.220461 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.221285 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.222886 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.223486 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.223611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.226705 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:07.226710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:07.226715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:07.226753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:07.229562 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.230256 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.230261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.230264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.230409 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.230773 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.230776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.230779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.231100 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:07.231242 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.231815 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.232040 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:06:07.232930 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.216214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:45561 (local address 127.27.186.130:56843)
0809 15:06:07.216884 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.216919 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.218508 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.219149 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.219157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.219680 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:07.220281 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.220306 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.221442 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.221448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.222734 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.222747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.223664 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.223669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.223855 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.224419 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:07.224443 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:07.226496 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:07.229683 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:07.229690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:07.229699 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:07.230134 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:07.230502 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:07.230505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:07.230507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:07.230670 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:07.231219 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:07.231225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:07.231479 (+ 254us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.231679 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.232011 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":264,"threads_started":1}
I20260809 15:06:07.234133 27658 heartbeater.cc:347] Connected to a master server at 127.27.186.190:45561
I20260809 15:06:07.234427 27658 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:07.235152 27658 heartbeater.cc:511] Master 127.27.186.190:45561 requested a full tablet report, sending...
I20260809 15:06:07.236542 27156 ts_manager.cc:194] Registered new tserver with Master: 0c1d046ff0ec4d3488035cee97c0fe2f (127.27.186.130:38851)
I20260809 15:06:07.237532 27156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.130@KRBTEST.COM'} at 127.27.186.130:56843
WARNING: no policy specified for kudu/127.27.186.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
Entry for principal kudu/127.27.186.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab.
I20260809 15:06:07.280427 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45561
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:07.588876 27666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:07.589118 27666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:07.589234 27666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:07.596127 27666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:07.596233 27666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:07.596377 27666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:06:07.605501 27666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.131.keytab
--principal=kudu/127.27.186.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:45561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:07.606580 27666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:07.607690 27666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:07.619724 27671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:07.619721 27672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:07.619750 27675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:07.620458 27666 server_base.cc:1034] running on GCE node
I20260809 15:06:07.620972 27666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:07.621767 27666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:07.622970 27666 hybrid_clock.cc:648] HybridClock initialized: now 1786287967622920 us; error 27 us; skew 500 ppm
I20260809 15:06:07.623246 27666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:07 dist-test-slave-q9h9 krb5kdc[27088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:07.626149 27666 init.cc:377] Logged in from keytab as kudu/127.27.186.131@KRBTEST.COM (short username kudu)
I20260809 15:06:07.630676 27666 webserver.cc:460] Webserver started at http://127.27.186.131:42789/ using document root <none> and password file <none>
I20260809 15:06:07.631390 27666 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:07.631490 27666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:07.631829 27666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:07.635169 27666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "5fa286f3428b4334a0ce4bd9f958e84d"
format_stamp: "Formatted at 2026-08-09 15:06:07 on dist-test-slave-q9h9"
server_key: "7e665b0c884050a879532f14b7da57b7"
server_key_iv: "6e84247dffa305663cca7558a340b6d5"
server_key_version: "encryptionkey@0"
I20260809 15:06:07.635895 27666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "5fa286f3428b4334a0ce4bd9f958e84d"
format_stamp: "Formatted at 2026-08-09 15:06:07 on dist-test-slave-q9h9"
server_key: "7e665b0c884050a879532f14b7da57b7"
server_key_iv: "6e84247dffa305663cca7558a340b6d5"
server_key_version: "encryptionkey@0"
I20260809 15:06:07.640605 27666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:06:07.643743 27685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:07.644876 27666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:06:07.645020 27666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "5fa286f3428b4334a0ce4bd9f958e84d"
format_stamp: "Formatted at 2026-08-09 15:06:07 on dist-test-slave-q9h9"
server_key: "7e665b0c884050a879532f14b7da57b7"
server_key_iv: "6e84247dffa305663cca7558a340b6d5"
server_key_version: "encryptionkey@0"
I20260809 15:06:07.645181 27666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:07.657124 27666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:07.660634 27666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:07.660898 27666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:07.661669 27666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:07.662976 27666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:07.663046 27666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:07.663131 27666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:07.663172 27666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:07.690032 27666 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:43123
I20260809 15:06:07.690096 27868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:43123 every 8 connection(s)
I20260809 15:06:07.691198 27666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:06:07.691506 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 27666
I20260809 15:06:07.691691 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:06:07.692240 28394 external_mini_cluster.cc:1428] Setting key 544c7126a26a7a825379053e9df07d9d
Aug 09 15:06:07 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.131@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:07.709607 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.693541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:53763 (local address 127.27.186.190:45561)
0809 15:06:07.693725 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.693731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.695069 (+ 1338us) server_negotiation.cc:407] Connection header received
0809 15:06:07.696853 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.696859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.696964 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.697047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:07.698440 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.699166 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.700602 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.701010 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.701116 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.704147 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:07.704151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:07.704154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:07.704186 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:07.706745 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.707479 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.707482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.707485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.707598 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.708015 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.708018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.708020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.708288 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:07.708418 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.709057 (+ 639us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.709280 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:06:07.710374 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.694142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:45561 (local address 127.27.186.131:53763)
0809 15:06:07.694924 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.694970 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.696615 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.697273 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.697280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.697741 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:07.698289 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.698306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.699318 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.699322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.700468 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.700480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.701212 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.701220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.701436 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.701925 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:07.701945 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:07.703960 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:07.706867 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:07.706875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:07.706889 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:07.707358 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:07.707697 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:07.707703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:07.707707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:07.707894 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:07.708410 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:07.708417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:07.708690 (+ 273us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.708914 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.709339 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":293,"threads_started":1}
I20260809 15:06:07.711438 27869 heartbeater.cc:347] Connected to a master server at 127.27.186.190:45561
I20260809 15:06:07.711722 27869 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:07.712369 27869 heartbeater.cc:511] Master 127.27.186.190:45561 requested a full tablet report, sending...
I20260809 15:06:07.713562 27156 ts_manager.cc:194] Registered new tserver with Master: 5fa286f3428b4334a0ce4bd9f958e84d (127.27.186.131:43123)
I20260809 15:06:07.714329 27156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.186.131@KRBTEST.COM'} at 127.27.186.131:53763
I20260809 15:06:07.718984 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:06:07.733345 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.722912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.27.186.190:45561)
0809 15:06:07.723082 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.723088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.723456 (+ 368us) server_negotiation.cc:407] Connection header received
0809 15:06:07.723564 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.723567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.723628 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.723698 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:07.724970 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.725620 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.726768 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.727140 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.727235 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.729229 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:07.729233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:07.729236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:07.729258 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:07.731352 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.731891 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.731894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.731896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.732007 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.732297 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.732300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.732302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.732554 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:07.732727 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.732970 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.733104 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:06:07.737001 27156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:06:07.743435 27156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:06:07.764148 27446 heartbeater.cc:503] Master 127.27.186.190:45561 was elected leader, sending a full tablet report...
I20260809 15:06:07.764655 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.754281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.27.186.131:43123)
0809 15:06:07.754943 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.754952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.755745 (+ 793us) server_negotiation.cc:407] Connection header received
0809 15:06:07.756061 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.756069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.756308 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.756490 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:07.757399 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.758968 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.761790 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.763172 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.763340 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.763505 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.764171 (+ 666us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.764339 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":497,"threads_started":1}
I20260809 15:06:07.765535 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.753966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:43123 (local address 127.0.0.1:57194)
0809 15:06:07.755580 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.755624 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.755813 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.756770 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.756777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.756805 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:07.757184 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.757198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.759140 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.759145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.761612 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.761627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.763688 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.763694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.763906 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.763963 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.764964 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.765140 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":1180,"threads_started":1}
I20260809 15:06:07.765794 27895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.754197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:43901 (local address 127.0.0.1:52484)
0809 15:06:07.756145 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.756288 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.756501 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.757358 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.757362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.757387 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:07.757812 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.757825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.759761 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.759766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.761833 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.761847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.763894 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.763900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.765071 (+ 1171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.765108 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.765200 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.765311 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"mutex_wait_us":4,"thread_start_us":1703,"threads_started":1}
I20260809 15:06:07.766705 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.754419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.27.186.129:43901)
0809 15:06:07.755044 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.755052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.756402 (+ 1350us) server_negotiation.cc:407] Connection header received
0809 15:06:07.756696 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.756704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.756924 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.757110 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:07.758036 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.759477 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.762002 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.763403 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.763581 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.763739 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.766248 (+ 2509us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.766414 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":295,"threads_started":1}
I20260809 15:06:07.769860 27766 tablet_service.cc:1467] Processing CreateTablet for tablet da9246f4613142599e8f919803dfff5d (DEFAULT_TABLE table=test-table [id=1300b48a4dfe47f8b830c7b097192574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:07.770678 27338 tablet_service.cc:1467] Processing CreateTablet for tablet da9246f4613142599e8f919803dfff5d (DEFAULT_TABLE table=test-table [id=1300b48a4dfe47f8b830c7b097192574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:07.771270 27766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9246f4613142599e8f919803dfff5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:07.772025 27338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9246f4613142599e8f919803dfff5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:07.774545 27895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.756221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38851 (local address 127.0.0.1:47028)
0809 15:06:07.765977 (+ 9756us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.765994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.766096 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.768100 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.768106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.768132 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:07.768284 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.768295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.770319 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.770324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.772117 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.772134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.773809 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.773816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.773999 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.774059 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.774152 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.774284 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9669,"thread_start_us":11384,"threads_started":1}
I20260809 15:06:07.775151 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.756515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.27.186.130:38851)
0809 15:06:07.767522 (+ 11007us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.767530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.767557 (+ 27us) server_negotiation.cc:407] Connection header received
0809 15:06:07.767654 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.767661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.767892 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.768445 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:07.768691 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.770167 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.772266 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.773610 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.773783 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.773938 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.774195 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.774355 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10803,"thread_start_us":10698,"threads_started":1}
I20260809 15:06:07.778700 27565 tablet_service.cc:1467] Processing CreateTablet for tablet da9246f4613142599e8f919803dfff5d (DEFAULT_TABLE table=test-table [id=1300b48a4dfe47f8b830c7b097192574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:07.780104 27565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9246f4613142599e8f919803dfff5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:07.783103 27898 tablet_bootstrap.cc:492] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Bootstrap starting.
I20260809 15:06:07.785784 27900 tablet_bootstrap.cc:492] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Bootstrap starting.
I20260809 15:06:07.787076 27898 tablet_bootstrap.cc:654] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:07.788341 27898 log.cc:824] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:07.789093 27900 tablet_bootstrap.cc:654] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:07.790624 27900 log.cc:824] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:07.791541 27898 tablet_bootstrap.cc:492] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: No bootstrap required, opened a new log
I20260809 15:06:07.791777 27898 ts_tablet_manager.cc:1397] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260809 15:06:07.792678 27910 tablet_bootstrap.cc:492] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Bootstrap starting.
I20260809 15:06:07.793609 27900 tablet_bootstrap.cc:492] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: No bootstrap required, opened a new log
I20260809 15:06:07.793848 27900 ts_tablet_manager.cc:1397] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260809 15:06:07.795310 27910 tablet_bootstrap.cc:654] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:07.796231 27910 log.cc:824] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:07.798955 27910 tablet_bootstrap.cc:492] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: No bootstrap required, opened a new log
I20260809 15:06:07.799175 27910 ts_tablet_manager.cc:1397] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:06:07.802165 27898 raft_consensus.cc:348] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.802574 27898 raft_consensus.cc:374] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:07.802671 27898 raft_consensus.cc:729] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100fac73d3e4b498d023ae887415016, State: Initialized, Role: FOLLOWER
I20260809 15:06:07.803263 27898 consensus_queue.cc:260] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.803726 27900 raft_consensus.cc:348] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.804126 27900 raft_consensus.cc:374] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:07.804245 27900 raft_consensus.cc:729] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa286f3428b4334a0ce4bd9f958e84d, State: Initialized, Role: FOLLOWER
I20260809 15:06:07.804798 27900 consensus_queue.cc:260] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.806530 27910 raft_consensus.cc:348] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.806828 27869 heartbeater.cc:503] Master 127.27.186.190:45561 was elected leader, sending a full tablet report...
I20260809 15:06:07.806901 27910 raft_consensus.cc:374] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:07.806998 27910 raft_consensus.cc:729] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d046ff0ec4d3488035cee97c0fe2f, State: Initialized, Role: FOLLOWER
I20260809 15:06:07.807482 27910 consensus_queue.cc:260] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.806576 27898 ts_tablet_manager.cc:1428] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260809 15:06:07.808622 27658 heartbeater.cc:503] Master 127.27.186.190:45561 was elected leader, sending a full tablet report...
I20260809 15:06:07.809186 27910 ts_tablet_manager.cc:1428] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260809 15:06:07.809994 27900 ts_tablet_manager.cc:1428] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260809 15:06:07.868453 27913 raft_consensus.cc:482] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:06:07.868682 27913 raft_consensus.cc:504] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.870210 27913 leader_election.cc:288] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa286f3428b4334a0ce4bd9f958e84d (127.27.186.131:43123), 0c1d046ff0ec4d3488035cee97c0fe2f (127.27.186.130:38851)
I20260809 15:06:07.876487 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.870647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:60411 (local address 127.27.186.131:43123)
0809 15:06:07.870843 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.870849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.871478 (+ 629us) server_negotiation.cc:407] Connection header received
0809 15:06:07.871649 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.871653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.871708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.871777 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:07.872341 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.873063 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.874882 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.875614 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.875754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.875827 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.876128 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.876249 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 15:06:07.876533 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.870603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:41439 (local address 127.27.186.130:38851)
0809 15:06:07.870801 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.870806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.871561 (+ 755us) server_negotiation.cc:407] Connection header received
0809 15:06:07.871715 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.871720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.871775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.871848 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:07.872436 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.873118 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.874932 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.875676 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.875812 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.875881 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.876164 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.876295 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:06:07.876744 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.870623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:43123 (local address 127.27.186.129:60411)
0809 15:06:07.871378 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.871393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.871522 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.871816 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.871820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.871845 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:07.872199 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.872209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.873190 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.873195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.874716 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.874728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.875841 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.875848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.875980 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.876003 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.876084 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.876206 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":3,"thread_start_us":422,"threads_started":1}
I20260809 15:06:07.876796 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.870658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:38851 (local address 127.27.186.129:41439)
0809 15:06:07.871459 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:07.871473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:07.871590 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:07.871874 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:07.871878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:07.871905 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:07.872280 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.872290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.873251 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.873256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.874775 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:07.874787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.875892 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.875899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:07.876019 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.876041 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 15:06:07.876121 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 15:06:07.876235 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":18,"thread_start_us":414,"threads_started":1}
I20260809 15:06:07.877575 27592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9246f4613142599e8f919803dfff5d" candidate_uuid: "1100fac73d3e4b498d023ae887415016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" is_pre_election: true
I20260809 15:06:07.877609 27797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9246f4613142599e8f919803dfff5d" candidate_uuid: "1100fac73d3e4b498d023ae887415016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" is_pre_election: true
I20260809 15:06:07.878008 27797 raft_consensus.cc:2454] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100fac73d3e4b498d023ae887415016 in term 0.
I20260809 15:06:07.877988 27592 raft_consensus.cc:2454] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100fac73d3e4b498d023ae887415016 in term 0.
I20260809 15:06:07.878598 27269 leader_election.cc:302] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100fac73d3e4b498d023ae887415016, 5fa286f3428b4334a0ce4bd9f958e84d; no voters:
I20260809 15:06:07.878935 27913 raft_consensus.cc:2781] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:06:07.879084 27913 raft_consensus.cc:482] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:06:07.879180 27913 raft_consensus.cc:3037] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:07.881196 27913 raft_consensus.cc:504] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.881708 27913 leader_election.cc:288] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa286f3428b4334a0ce4bd9f958e84d (127.27.186.131:43123), 0c1d046ff0ec4d3488035cee97c0fe2f (127.27.186.130:38851)
I20260809 15:06:07.882113 27797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9246f4613142599e8f919803dfff5d" candidate_uuid: "1100fac73d3e4b498d023ae887415016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa286f3428b4334a0ce4bd9f958e84d"
I20260809 15:06:07.882193 27592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9246f4613142599e8f919803dfff5d" candidate_uuid: "1100fac73d3e4b498d023ae887415016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f"
I20260809 15:06:07.882319 27797 raft_consensus.cc:3037] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:07.882376 27592 raft_consensus.cc:3037] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:07.885432 27592 raft_consensus.cc:2454] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100fac73d3e4b498d023ae887415016 in term 1.
I20260809 15:06:07.885432 27797 raft_consensus.cc:2454] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100fac73d3e4b498d023ae887415016 in term 1.
I20260809 15:06:07.885818 27271 leader_election.cc:302] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1d046ff0ec4d3488035cee97c0fe2f, 1100fac73d3e4b498d023ae887415016; no voters:
I20260809 15:06:07.886091 27913 raft_consensus.cc:2781] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:07.886498 27913 raft_consensus.cc:686] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 1 LEADER]: Becoming Leader. State: Replica: 1100fac73d3e4b498d023ae887415016, State: Running, Role: LEADER
I20260809 15:06:07.887179 27913 consensus_queue.cc:237] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } }
I20260809 15:06:07.891361 27156 catalog_manager.cc:5495] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1100fac73d3e4b498d023ae887415016 (127.27.186.129). New cstate: current_term: 1 leader_uuid: "1100fac73d3e4b498d023ae887415016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1100fac73d3e4b498d023ae887415016" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 43901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 } health_report { overall_health: UNKNOWN } } }
I20260809 15:06:07.926899 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.921864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.27.186.129:43901)
0809 15:06:07.922076 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.922082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.922178 (+ 96us) server_negotiation.cc:407] Connection header received
0809 15:06:07.922369 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.922374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.922462 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.922577 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:07.922942 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.923616 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.924941 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.925331 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.925435 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.925893 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:07.926362 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:07.926486 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.926592 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.926666 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:06:07.933755 27592 raft_consensus.cc:1261] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 1 FOLLOWER]: Refusing update from remote peer 1100fac73d3e4b498d023ae887415016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:07.933789 27797 raft_consensus.cc:1261] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 1 FOLLOWER]: Refusing update from remote peer 1100fac73d3e4b498d023ae887415016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:07.934718 27913 consensus_queue.cc:1035] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa286f3428b4334a0ce4bd9f958e84d" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:07.934995 27924 consensus_queue.cc:1035] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d046ff0ec4d3488035cee97c0fe2f" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:06:07.943114 27871 tablet.cc:2369] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:07.944912 27928 mvcc.cc:205] Tried to move back new op lower bound from 7316635516648456192 to 7316635516469248000. Current Snapshot: MvccSnapshot[applied={T|T < 7316635516648456192}]
I20260809 15:06:07.948374 27930 mvcc.cc:205] Tried to move back new op lower bound from 7316635516648456192 to 7316635516469248000. Current Snapshot: MvccSnapshot[applied={T|T < 7316635516648456192 or (T in {7316635516648456192})}]
I20260809 15:06:07.957088 27154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:07.957407 27154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:07.960956 27154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e005c3ec276249409afb0e1f25f50a0d: Sending DeleteTablet for 3 replicas of tablet da9246f4613142599e8f919803dfff5d
I20260809 15:06:07.961956 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet da9246f4613142599e8f919803dfff5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:07 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:57194
I20260809 15:06:07.962265 27565 tablet_service.cc:1514] Processing DeleteTablet for tablet da9246f4613142599e8f919803dfff5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:07 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:47028
I20260809 15:06:07.962823 27941 tablet_replica.cc:330] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: stopping tablet replica
I20260809 15:06:07.962785 27338 tablet_service.cc:1514] Processing DeleteTablet for tablet da9246f4613142599e8f919803dfff5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:07 UTC) from {username='kudu', principal='kudu/127.27.186.190@KRBTEST.COM'} at 127.0.0.1:52484
I20260809 15:06:07.963038 27942 tablet_replica.cc:330] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: stopping tablet replica
I20260809 15:06:07.963156 27941 raft_consensus.cc:2229] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:07.963388 27942 raft_consensus.cc:2229] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:07.963559 27941 raft_consensus.cc:2258] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:07.963789 27942 raft_consensus.cc:2258] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:07.964658 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 27243
I20260809 15:06:07.965468 27941 ts_tablet_manager.cc:1905] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:07.966005 27942 ts_tablet_manager.cc:1905] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:07.972187 27941 ts_tablet_manager.cc:1918] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:07.972333 27941 log.cc:1196] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/da9246f4613142599e8f919803dfff5d
I20260809 15:06:07.972538 27942 ts_tablet_manager.cc:1918] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:07.972687 27942 log.cc:1196] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/da9246f4613142599e8f919803dfff5d
I20260809 15:06:07.972843 27941 ts_tablet_manager.cc:1939] T da9246f4613142599e8f919803dfff5d P 5fa286f3428b4334a0ce4bd9f958e84d: Deleting consensus metadata
I20260809 15:06:07.973196 27942 ts_tablet_manager.cc:1939] T da9246f4613142599e8f919803dfff5d P 0c1d046ff0ec4d3488035cee97c0fe2f: Deleting consensus metadata
I20260809 15:06:07.974469 27137 catalog_manager.cc:4841] TS 5fa286f3428b4334a0ce4bd9f958e84d (127.27.186.131:43123): tablet da9246f4613142599e8f919803dfff5d (table test-table [id=1300b48a4dfe47f8b830c7b097192574]) successfully deleted
I20260809 15:06:07.974776 27139 catalog_manager.cc:4841] TS 0c1d046ff0ec4d3488035cee97c0fe2f (127.27.186.130:38851): tablet da9246f4613142599e8f919803dfff5d (table test-table [id=1300b48a4dfe47f8b830c7b097192574]) successfully deleted
I20260809 15:06:07.975703 27945 tablet_replica.cc:330] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: stopping tablet replica
I20260809 15:06:07.976011 27945 raft_consensus.cc:2229] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:06:07.976459 27945 raft_consensus.cc:2258] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:07.978168 27945 ts_tablet_manager.cc:1905] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:07.983460 27945 ts_tablet_manager.cc:1918] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:07.983610 27945 log.cc:1196] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/da9246f4613142599e8f919803dfff5d
I20260809 15:06:07.984123 27945 ts_tablet_manager.cc:1939] T da9246f4613142599e8f919803dfff5d P 1100fac73d3e4b498d023ae887415016: Deleting consensus metadata
Aug 09 15:06:07 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.129@KRBTEST.COM
I20260809 15:06:07.985749 27139 catalog_manager.cc:4841] TS 1100fac73d3e4b498d023ae887415016 (127.27.186.129:43901): tablet da9246f4613142599e8f919803dfff5d (table test-table [id=1300b48a4dfe47f8b830c7b097192574]) successfully deleted
I20260809 15:06:07.992705 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:07.967061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.27.186.129:43901)
0809 15:06:07.967208 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:07.967225 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:07.975122 (+ 7897us) server_negotiation.cc:407] Connection header received
0809 15:06:07.975329 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:07.975334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:07.975429 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:07.975539 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:07.977246 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.978429 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.980217 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:07.980897 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:07.981019 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:07.986123 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:07.986137 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:07.986153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:07.986201 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:07.989900 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.990631 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.990639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.990643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.990814 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:07.991161 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:07.991166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:07.991170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:07.991608 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:07.991901 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:07.992098 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 15:06:07.992346 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:06:07.993391 27438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:08.087109 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 27243
I20260809 15:06:08.116905 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 27461
Aug 09 15:06:08 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.130@KRBTEST.COM
I20260809 15:06:08.131247 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:08.117613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.27.186.130:38851)
0809 15:06:08.117789 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:08.117795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:08.117849 (+ 54us) server_negotiation.cc:407] Connection header received
0809 15:06:08.118052 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:08.118057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:08.118121 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:08.118206 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:08.119498 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:08.120218 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:08.121550 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:08.121935 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:08.122174 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:08.126035 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:08.126054 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:08.126073 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:08.126120 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:08.128624 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:08.129241 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:08.129248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:08.129251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:08.129384 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:08.129729 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:08.129733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:08.129735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:08.130152 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:08.130442 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:08.130724 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 15:06:08.130925 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 15:06:08.131878 27653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:08.232172 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 27461
I20260809 15:06:08.255810 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 27666
Aug 09 15:06:08 dist-test-slave-q9h9 krb5kdc[27088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.131@KRBTEST.COM
I20260809 15:06:08.270954 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:08.256643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.27.186.131:43123)
0809 15:06:08.256853 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:08.256858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:08.256878 (+ 20us) server_negotiation.cc:407] Connection header received
0809 15:06:08.257054 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:08.257060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:08.257132 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:08.257226 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:08.258725 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:08.259527 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:08.260916 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:08.261338 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:08.261449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:08.265877 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:08.265893 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:08.265910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:08.265949 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:08.268615 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:08.269198 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:08.269204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:08.269206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:08.269363 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:08.269636 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:08.269639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:08.269641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:08.269974 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:08.270318 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:08.270495 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 15:06:08.270658 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 15:06:08.271615 27864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:08.357672 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 27666
I20260809 15:06:08.379727 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 27103
I20260809 15:06:08.380787 27194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:08.485394 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 27103
2026-08-09T15:06:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4171 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T15:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:08Z Disabled control of system clock
I20260809 15:06:08.568499 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42055
--webserver_interface=127.27.186.190
--webserver_port=0
--builtin_ntp_servers=127.27.186.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 15:06:08.877674 27965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:08.877908 27965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:08.877990 27965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:08.884716 27965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:08.884801 27965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:08.884845 27965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:08.884902 27965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:08.884944 27965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:08.893347 27965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:42055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:08.894316 27965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:08.895823 27965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:08.906304 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:08.906339 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:08.906389 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:08.907397 27965 server_base.cc:1034] running on GCE node
I20260809 15:06:08.908038 27965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:08.909178 27965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:08.910370 27965 hybrid_clock.cc:648] HybridClock initialized: now 1786287968910335 us; error 49 us; skew 500 ppm
I20260809 15:06:08.910676 27965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:08.912686 27965 webserver.cc:460] Webserver started at http://127.27.186.190:46003/ using document root <none> and password file <none>
I20260809 15:06:08.913290 27965 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:08.913390 27965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:08.913669 27965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:08.916312 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "0a77ef758294403188866da8290b47db"
format_stamp: "Formatted at 2026-08-09 15:06:08 on dist-test-slave-q9h9"
server_key: "16d99a23f8c938c5887d8a3afe0afe7b"
server_key_iv: "b993a44a8d1d2ee03ba8f1c7a07aa410"
server_key_version: "encryptionkey@0"
I20260809 15:06:08.917032 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "0a77ef758294403188866da8290b47db"
format_stamp: "Formatted at 2026-08-09 15:06:08 on dist-test-slave-q9h9"
server_key: "16d99a23f8c938c5887d8a3afe0afe7b"
server_key_iv: "b993a44a8d1d2ee03ba8f1c7a07aa410"
server_key_version: "encryptionkey@0"
I20260809 15:06:08.921847 27965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:06:08.924990 27985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:08.926216 27965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:08.926364 27965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "0a77ef758294403188866da8290b47db"
format_stamp: "Formatted at 2026-08-09 15:06:08 on dist-test-slave-q9h9"
server_key: "16d99a23f8c938c5887d8a3afe0afe7b"
server_key_iv: "b993a44a8d1d2ee03ba8f1c7a07aa410"
server_key_version: "encryptionkey@0"
I20260809 15:06:08.926541 27965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:08.938299 27965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:08.938946 27965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:08.939191 27965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:08.955506 27965 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:42055
I20260809 15:06:08.955536 28071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:42055 every 8 connection(s)
I20260809 15:06:08.956602 27965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:08.957572 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 27965
I20260809 15:06:08.957757 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:08.958303 28394 external_mini_cluster.cc:1428] Setting key 3cf3b009d2e312efa257a010d420d451
I20260809 15:06:08.960886 28073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:08.972693 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:08.960041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.27.186.190:42055)
0809 15:06:08.961022 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:08.961036 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:08.961083 (+ 47us) server_negotiation.cc:407] Connection header received
0809 15:06:08.963264 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:08.963290 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:08.963867 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:08.964280 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:08.964778 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:08.966705 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:08.968005 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:08.968965 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:08.969159 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:08.970149 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:08.970163 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:08.970180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:08.970854 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:08.970945 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:08.970973 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:08.971088 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:08.971262 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 15:06:08.971562 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":329,"threads_started":1}
I20260809 15:06:08.973870 28073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db: Bootstrap starting.
I20260809 15:06:08.976653 28073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:08.977587 28073 log.cc:824] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:08.980049 28073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db: No bootstrap required, opened a new log
I20260809 15:06:08.986824 28073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a77ef758294403188866da8290b47db" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42055 } }
I20260809 15:06:08.987125 28073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:08.987236 28073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a77ef758294403188866da8290b47db, State: Initialized, Role: FOLLOWER
I20260809 15:06:08.987744 28073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a77ef758294403188866da8290b47db" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42055 } }
I20260809 15:06:08.987952 28073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:08.988070 28073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:08.988229 28073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:08.990288 28073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a77ef758294403188866da8290b47db" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42055 } }
I20260809 15:06:08.990723 28073 leader_election.cc:302] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a77ef758294403188866da8290b47db; no voters:
I20260809 15:06:08.991243 28073 leader_election.cc:288] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:08.991350 28079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:08.991986 28079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [term 1 LEADER]: Becoming Leader. State: Replica: 0a77ef758294403188866da8290b47db, State: Running, Role: LEADER
I20260809 15:06:08.992446 28079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a77ef758294403188866da8290b47db" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42055 } }
I20260809 15:06:08.992952 28073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:08.995240 28082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a77ef758294403188866da8290b47db. Latest consensus state: current_term: 1 leader_uuid: "0a77ef758294403188866da8290b47db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a77ef758294403188866da8290b47db" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42055 } } }
I20260809 15:06:08.995263 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a77ef758294403188866da8290b47db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a77ef758294403188866da8290b47db" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42055 } } }
I20260809 15:06:08.995800 28082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:08.995805 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:08.999099 28096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:09.003073 28096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:09.010466 28096 catalog_manager.cc:1347] Generated new cluster ID: d06b753848a34049a25df18ceeddecb7
I20260809 15:06:09.010562 28096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:09.017077 28096 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:09.018674 28096 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:09.029770 28096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db: Generated new TSK 0
I20260809 15:06:09.030649 28096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:09.041667 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.129:0
--local_ip_for_outbound_sockets=127.27.186.129
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42055
--builtin_ntp_servers=127.27.186.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 15:06:09.369774 28107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:09.370030 28107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:09.370153 28107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:09.377032 28107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:09.377116 28107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:09.377259 28107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.129
I20260809 15:06:09.385732 28107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.186.129
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.129
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:09.386795 28107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:09.388456 28107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:09.400892 28114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:09.400898 28115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:09.400938 28119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:09.401760 28107 server_base.cc:1034] running on GCE node
I20260809 15:06:09.402364 28107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:09.403193 28107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:09.404376 28107 hybrid_clock.cc:648] HybridClock initialized: now 1786287969404328 us; error 45 us; skew 500 ppm
I20260809 15:06:09.404647 28107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:09.406759 28107 webserver.cc:460] Webserver started at http://127.27.186.129:39567/ using document root <none> and password file <none>
I20260809 15:06:09.407356 28107 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:09.407449 28107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:09.407742 28107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:09.410372 28107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data/instance:
uuid: "f85dd1bfd7424e33a6b5a3042be904ed"
format_stamp: "Formatted at 2026-08-09 15:06:09 on dist-test-slave-q9h9"
server_key: "4aa61ac0d6390b2c251d15b99d2ddd57"
server_key_iv: "c8d1bff35c6777e9a32f91c5f305e5b9"
server_key_version: "encryptionkey@0"
I20260809 15:06:09.411096 28107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance:
uuid: "f85dd1bfd7424e33a6b5a3042be904ed"
format_stamp: "Formatted at 2026-08-09 15:06:09 on dist-test-slave-q9h9"
server_key: "4aa61ac0d6390b2c251d15b99d2ddd57"
server_key_iv: "c8d1bff35c6777e9a32f91c5f305e5b9"
server_key_version: "encryptionkey@0"
I20260809 15:06:09.415729 28107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 15:06:09.418901 28132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:09.420038 28107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:06:09.420176 28107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal
uuid: "f85dd1bfd7424e33a6b5a3042be904ed"
format_stamp: "Formatted at 2026-08-09 15:06:09 on dist-test-slave-q9h9"
server_key: "4aa61ac0d6390b2c251d15b99d2ddd57"
server_key_iv: "c8d1bff35c6777e9a32f91c5f305e5b9"
server_key_version: "encryptionkey@0"
I20260809 15:06:09.420358 28107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:09.440409 28107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:09.441078 28107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:09.441339 28107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:09.442245 28107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:09.443575 28107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:09.443670 28107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:09.443773 28107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:09.443841 28107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:09.473474 28107 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.129:38541
I20260809 15:06:09.473537 28312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.129:38541 every 8 connection(s)
I20260809 15:06:09.474612 28107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/data/info.pb
I20260809 15:06:09.479358 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28107
I20260809 15:06:09.479526 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal/instance
I20260809 15:06:09.480033 28394 external_mini_cluster.cc:1428] Setting key 608c30eafc1321060f373f93b707f77d
I20260809 15:06:09.482779 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.130:0
--local_ip_for_outbound_sockets=127.27.186.130
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42055
--builtin_ntp_servers=127.27.186.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 15:06:09.487003 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:09.476805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.129:38511 (local address 127.27.186.190:42055)
0809 15:06:09.477257 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:09.477263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:09.478297 (+ 1034us) server_negotiation.cc:407] Connection header received
0809 15:06:09.480353 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:09.480357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:09.480422 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:09.480515 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:09.481564 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.482416 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.484525 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.484932 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.485026 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:09.486334 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:09.486339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:09.486343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:09.486360 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:09.486397 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:09.486409 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:09.486475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:09.486675 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 15:06:09.486757 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":208,"threads_started":1}
I20260809 15:06:09.488152 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:09.477309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42055 (local address 127.27.186.129:38511)
0809 15:06:09.478165 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:09.478204 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:09.480081 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:09.480721 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:09.480728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:09.480827 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:09.481408 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:09.481426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.482581 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.482587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:09.484389 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:09.484406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.485135 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.485141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:09.485392 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:09.486031 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:06:09.486036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:09.486084 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:06:09.486170 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:09.486492 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:09.486502 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:09.486508 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:06:09.486661 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 15:06:09.486987 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":252,"threads_started":1}
I20260809 15:06:09.489490 28313 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42055
I20260809 15:06:09.489770 28313 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:09.490574 28313 heartbeater.cc:511] Master 127.27.186.190:42055 requested a full tablet report, sending...
I20260809 15:06:09.492479 28016 ts_manager.cc:194] Registered new tserver with Master: f85dd1bfd7424e33a6b5a3042be904ed (127.27.186.129:38541)
I20260809 15:06:09.493922 28016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.129:38511
W20260809 15:06:09.800985 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:09.801213 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:09.801291 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:09.808071 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:09.808151 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:09.808254 28321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.130
I20260809 15:06:09.816677 28321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.186.130
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.130
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:09.817621 28321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:09.819418 28321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:09.831329 28328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:09.831373 28330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:09.831413 28333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:09.835126 28321 server_base.cc:1034] running on GCE node
I20260809 15:06:09.835624 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:09.836395 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:09.837587 28321 hybrid_clock.cc:648] HybridClock initialized: now 1786287969837551 us; error 28 us; skew 500 ppm
I20260809 15:06:09.837862 28321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:09.839867 28321 webserver.cc:460] Webserver started at http://127.27.186.130:33473/ using document root <none> and password file <none>
I20260809 15:06:09.840478 28321 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:09.840564 28321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:09.840821 28321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:09.843408 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data/instance:
uuid: "edbe493b25124cb6a2722ffa8197f2f8"
format_stamp: "Formatted at 2026-08-09 15:06:09 on dist-test-slave-q9h9"
server_key: "28ab1d6a5694fce8d8e5a71dff229871"
server_key_iv: "23cc5b70c35240df9486674a21d72906"
server_key_version: "encryptionkey@0"
I20260809 15:06:09.844116 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance:
uuid: "edbe493b25124cb6a2722ffa8197f2f8"
format_stamp: "Formatted at 2026-08-09 15:06:09 on dist-test-slave-q9h9"
server_key: "28ab1d6a5694fce8d8e5a71dff229871"
server_key_iv: "23cc5b70c35240df9486674a21d72906"
server_key_version: "encryptionkey@0"
I20260809 15:06:09.848670 28321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 15:06:09.851853 28343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:09.852977 28321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:06:09.853106 28321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal
uuid: "edbe493b25124cb6a2722ffa8197f2f8"
format_stamp: "Formatted at 2026-08-09 15:06:09 on dist-test-slave-q9h9"
server_key: "28ab1d6a5694fce8d8e5a71dff229871"
server_key_iv: "23cc5b70c35240df9486674a21d72906"
server_key_version: "encryptionkey@0"
I20260809 15:06:09.853291 28321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:09.870126 28321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:09.870842 28321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:09.871099 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:09.871927 28321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:09.873190 28321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:09.873281 28321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:09.873381 28321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:09.873448 28321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:09.902633 28321 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.130:39935
I20260809 15:06:09.902741 28529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.130:39935 every 8 connection(s)
I20260809 15:06:09.903761 28321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/data/info.pb
I20260809 15:06:09.911211 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28321
I20260809 15:06:09.911355 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal/instance
I20260809 15:06:09.911749 28394 external_mini_cluster.cc:1428] Setting key 028137407cbed6c2f2cf8d37d508b25b
I20260809 15:06:09.914647 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.186.131:0
--local_ip_for_outbound_sockets=127.27.186.131
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42055
--builtin_ntp_servers=127.27.186.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 15:06:09.914901 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:09.905944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.130:60035 (local address 127.27.186.190:42055)
0809 15:06:09.906134 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:09.906139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:09.907315 (+ 1176us) server_negotiation.cc:407] Connection header received
0809 15:06:09.908887 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:09.908891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:09.908955 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:09.909027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:09.909905 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.910633 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.912167 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.912694 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.912787 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:09.914129 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:09.914135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:09.914139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:09.914158 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:09.914194 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:09.914210 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:09.914317 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:09.914492 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 15:06:09.914579 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:06:09.915657 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:09.906382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42055 (local address 127.27.186.130:60035)
0809 15:06:09.907162 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:09.907213 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:09.908665 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:09.909205 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:09.909212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:09.909292 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:09.909781 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:09.909797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.910739 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.910743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:09.912025 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:09.912037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:09.912903 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:09.912910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:09.913167 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:09.913832 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:06:09.913834 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:09.913868 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:06:09.913935 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:09.914310 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:09.914316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:09.914322 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 15:06:09.914452 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 15:06:09.914695 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":245,"threads_started":1}
I20260809 15:06:09.916882 28533 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42055
I20260809 15:06:09.917155 28533 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:09.917922 28533 heartbeater.cc:511] Master 127.27.186.190:42055 requested a full tablet report, sending...
I20260809 15:06:09.919338 28016 ts_manager.cc:194] Registered new tserver with Master: edbe493b25124cb6a2722ffa8197f2f8 (127.27.186.130:39935)
I20260809 15:06:09.920259 28016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.130:60035
W20260809 15:06:10.234892 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:10.235148 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:10.235276 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:10.242324 28536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:10.242416 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:10.242556 28536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.186.131
I20260809 15:06:10.251120 28536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.186.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.186.131
--webserver_port=0
--tserver_master_addrs=127.27.186.190:42055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.186.131
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:10.252229 28536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:10.253860 28536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:10.266836 28545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:10.266768 28546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:10.266767 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:10.267426 28536 server_base.cc:1034] running on GCE node
I20260809 15:06:10.268080 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:10.268936 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:10.270098 28536 hybrid_clock.cc:648] HybridClock initialized: now 1786287970270055 us; error 33 us; skew 500 ppm
I20260809 15:06:10.270393 28536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:06:10.272472 28536 webserver.cc:460] Webserver started at http://127.27.186.131:45495/ using document root <none> and password file <none>
I20260809 15:06:10.273104 28536 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:10.273208 28536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:10.273499 28536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:10.276266 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data/instance:
uuid: "2eea1fbf21074c429b56e2c728f9e72a"
format_stamp: "Formatted at 2026-08-09 15:06:10 on dist-test-slave-q9h9"
server_key: "05791d9dd8295abacc179afae2b7ec8f"
server_key_iv: "4e326d68c3e8eed257084eb3033f31e9"
server_key_version: "encryptionkey@0"
I20260809 15:06:10.277010 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance:
uuid: "2eea1fbf21074c429b56e2c728f9e72a"
format_stamp: "Formatted at 2026-08-09 15:06:10 on dist-test-slave-q9h9"
server_key: "05791d9dd8295abacc179afae2b7ec8f"
server_key_iv: "4e326d68c3e8eed257084eb3033f31e9"
server_key_version: "encryptionkey@0"
I20260809 15:06:10.281692 28536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260809 15:06:10.284860 28556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:10.286077 28536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:10.286221 28536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal
uuid: "2eea1fbf21074c429b56e2c728f9e72a"
format_stamp: "Formatted at 2026-08-09 15:06:10 on dist-test-slave-q9h9"
server_key: "05791d9dd8295abacc179afae2b7ec8f"
server_key_iv: "4e326d68c3e8eed257084eb3033f31e9"
server_key_version: "encryptionkey@0"
I20260809 15:06:10.286401 28536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:10.300367 28536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:10.301044 28536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:10.301306 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:10.302228 28536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:06:10.304738 28536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:06:10.304824 28536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:10.304901 28536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:06:10.304955 28536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:10.334527 28536 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.131:34281
I20260809 15:06:10.334589 28728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.131:34281 every 8 connection(s)
I20260809 15:06:10.335785 28536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/data/info.pb
I20260809 15:06:10.342869 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28536
I20260809 15:06:10.343019 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal/instance
I20260809 15:06:10.343540 28394 external_mini_cluster.cc:1428] Setting key 2f5337b7f2037090e63db0d0c89dc6a5
I20260809 15:06:10.347508 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.338082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:54285 (local address 127.27.186.190:42055)
0809 15:06:10.338282 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.338288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.339534 (+ 1246us) server_negotiation.cc:407] Connection header received
0809 15:06:10.341209 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.341213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.341276 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.341352 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:10.342395 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.343168 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.344897 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.345409 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.345516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.346739 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:10.346744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:10.346747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:10.346763 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:10.346800 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:10.346812 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:10.346884 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.347099 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.347188 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:06:10.348356 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.338560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:42055 (local address 127.27.186.131:54285)
0809 15:06:10.339392 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:10.339426 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:10.340973 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:10.341552 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:10.341559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:10.341639 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:10.342240 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.342260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.343333 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.343338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.344768 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.344783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.345615 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.345624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.345830 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.346456 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 15:06:10.346459 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:10.346495 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 15:06:10.346563 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:10.346897 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:10.346909 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:10.346916 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 15:06:10.347057 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 15:06:10.347413 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":258,"threads_started":1}
I20260809 15:06:10.349457 28729 heartbeater.cc:347] Connected to a master server at 127.27.186.190:42055
I20260809 15:06:10.349725 28729 heartbeater.cc:464] Registering TS with master...
I20260809 15:06:10.350481 28729 heartbeater.cc:511] Master 127.27.186.190:42055 requested a full tablet report, sending...
I20260809 15:06:10.351761 28016 ts_manager.cc:194] Registered new tserver with Master: 2eea1fbf21074c429b56e2c728f9e72a (127.27.186.131:34281)
I20260809 15:06:10.352515 28016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.186.131:54285
I20260809 15:06:10.358222 28394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 15:06:10.367511 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.362050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.27.186.190:42055)
0809 15:06:10.362185 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.362191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.362619 (+ 428us) server_negotiation.cc:407] Connection header received
0809 15:06:10.362775 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.362781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.362839 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.362916 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:10.363384 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.364122 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.365398 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.365796 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.365903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.366864 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:10.366869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:10.366873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:10.366889 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:10.366919 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:10.366931 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:10.367002 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.367128 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.367216 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 15:06:10.370818 28016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:06:10.377380 28016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:06:10.398788 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.388254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.27.186.129:38541)
0809 15:06:10.390368 (+ 2114us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.390377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.390406 (+ 29us) server_negotiation.cc:407] Connection header received
0809 15:06:10.390501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.390507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.390758 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.391136 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:10.391995 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.393471 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.396249 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.397666 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.397814 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.398046 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.398274 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.398452 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1967,"thread_start_us":1838,"threads_started":1}
I20260809 15:06:10.399335 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.388038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:38541 (local address 127.0.0.1:50680)
0809 15:06:10.388871 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:10.388907 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:10.389064 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:10.390998 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:10.391009 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:10.391042 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:10.391454 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.391468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.393605 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.393610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.396052 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.396067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.397835 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.397841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.398069 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.398119 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 15:06:10.398218 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 15:06:10.398452 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":27,"thread_start_us":447,"threads_started":1}
I20260809 15:06:10.399842 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.390328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39935 (local address 127.0.0.1:42656)
0809 15:06:10.391780 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:10.391806 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:10.391993 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:10.392636 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:10.392644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:10.392683 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:10.393123 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.393135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.394615 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.394624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.396884 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.396896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.399050 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.399057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.399291 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.399322 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 15:06:10.399387 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 15:06:10.399527 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"mutex_wait_us":139,"thread_start_us":659,"threads_started":1}
I20260809 15:06:10.400033 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.388887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.27.186.131:34281)
0809 15:06:10.390973 (+ 2086us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.390990 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.391018 (+ 28us) server_negotiation.cc:407] Connection header received
0809 15:06:10.391099 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.391106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.391313 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.392359 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:10.392987 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.394425 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.397154 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.398523 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.398712 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.399472 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.399533 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.399689 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1922,"thread_start_us":1777,"threads_started":1}
I20260809 15:06:10.400588 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.388294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.131:34281 (local address 127.0.0.1:52146)
0809 15:06:10.389761 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:10.389779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:10.389886 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:10.392365 (+ 2479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:10.392370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:10.392395 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:10.392780 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.392802 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.394567 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.394573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.396975 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.396991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.399062 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.399067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.399270 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.399303 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 15:06:10.399384 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 15:06:10.400276 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":883,"thread_start_us":1157,"threads_started":1}
I20260809 15:06:10.400712 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.390774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.27.186.130:39935)
0809 15:06:10.391996 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.392004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.392030 (+ 26us) server_negotiation.cc:407] Connection header received
0809 15:06:10.392125 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.392132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.392365 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.392536 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:10.393281 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.394456 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.397092 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.398522 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.398705 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.398905 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.400205 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.400356 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":939,"threads_started":1}
I20260809 15:06:10.402940 28208 tablet_service.cc:1467] Processing CreateTablet for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 (DEFAULT_TABLE table=test-table [id=43e10425570140618def78bdd15d0177]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:10.404032 28208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:10.403972 28630 tablet_service.cc:1467] Processing CreateTablet for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 (DEFAULT_TABLE table=test-table [id=43e10425570140618def78bdd15d0177]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:10.403973 28431 tablet_service.cc:1467] Processing CreateTablet for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 (DEFAULT_TABLE table=test-table [id=43e10425570140618def78bdd15d0177]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 15:06:10.405354 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:10.405421 28630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:10.414772 28756 tablet_bootstrap.cc:492] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Bootstrap starting.
I20260809 15:06:10.418371 28756 tablet_bootstrap.cc:654] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:10.419319 28756 log.cc:824] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:10.421875 28756 tablet_bootstrap.cc:492] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: No bootstrap required, opened a new log
I20260809 15:06:10.422137 28756 ts_tablet_manager.cc:1397] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 15:06:10.425084 28757 tablet_bootstrap.cc:492] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Bootstrap starting.
I20260809 15:06:10.428419 28757 tablet_bootstrap.cc:654] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:10.428572 28758 tablet_bootstrap.cc:492] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Bootstrap starting.
I20260809 15:06:10.429549 28757 log.cc:824] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:10.430953 28756 raft_consensus.cc:348] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.431380 28756 raft_consensus.cc:374] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:10.431492 28758 tablet_bootstrap.cc:654] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:10.431481 28756 raft_consensus.cc:729] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85dd1bfd7424e33a6b5a3042be904ed, State: Initialized, Role: FOLLOWER
I20260809 15:06:10.432081 28756 consensus_queue.cc:260] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.432461 28758 log.cc:824] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:10.432677 28757 tablet_bootstrap.cc:492] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: No bootstrap required, opened a new log
I20260809 15:06:10.432933 28757 ts_tablet_manager.cc:1397] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 15:06:10.433341 28313 heartbeater.cc:503] Master 127.27.186.190:42055 was elected leader, sending a full tablet report...
I20260809 15:06:10.433826 28756 ts_tablet_manager.cc:1428] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260809 15:06:10.437350 28758 tablet_bootstrap.cc:492] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: No bootstrap required, opened a new log
I20260809 15:06:10.437592 28758 ts_tablet_manager.cc:1397] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260809 15:06:10.441009 28757 raft_consensus.cc:348] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.441313 28757 raft_consensus.cc:374] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:10.441401 28757 raft_consensus.cc:729] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eea1fbf21074c429b56e2c728f9e72a, State: Initialized, Role: FOLLOWER
I20260809 15:06:10.441857 28757 consensus_queue.cc:260] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.442826 28729 heartbeater.cc:503] Master 127.27.186.190:42055 was elected leader, sending a full tablet report...
I20260809 15:06:10.443235 28757 ts_tablet_manager.cc:1428] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260809 15:06:10.445556 28758 raft_consensus.cc:348] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.445955 28758 raft_consensus.cc:374] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:10.446070 28758 raft_consensus.cc:729] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbe493b25124cb6a2722ffa8197f2f8, State: Initialized, Role: FOLLOWER
I20260809 15:06:10.446597 28758 consensus_queue.cc:260] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.447578 28533 heartbeater.cc:503] Master 127.27.186.190:42055 was elected leader, sending a full tablet report...
I20260809 15:06:10.448038 28758 ts_tablet_manager.cc:1428] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
W20260809 15:06:10.476550 28315 tablet.cc:2369] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 15:06:10.587318 28730 tablet.cc:2369] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:10.615585 28766 raft_consensus.cc:482] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:06:10.615852 28766 raft_consensus.cc:504] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.617412 28766 leader_election.cc:288] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f85dd1bfd7424e33a6b5a3042be904ed (127.27.186.129:38541), edbe493b25124cb6a2722ffa8197f2f8 (127.27.186.130:39935)
I20260809 15:06:10.623587 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.618117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:47925 (local address 127.27.186.130:39935)
0809 15:06:10.618300 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.618306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.618325 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:06:10.618399 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.618404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.618480 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.618579 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:10.619087 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.619967 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.621991 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.622790 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.622932 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.623002 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.623234 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.623359 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:06:10.623638 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.617632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.130:39935 (local address 127.27.186.131:47925)
0809 15:06:10.617885 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:10.617903 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:10.618052 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:10.618687 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:10.618691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:10.618718 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:10.618938 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.618949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.620109 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.620115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.621843 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.621855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.622937 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.622943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.623094 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.623118 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 15:06:10.623195 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 15:06:10.623335 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1}
I20260809 15:06:10.624017 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.617750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.186.131:56193 (local address 127.27.186.129:38541)
0809 15:06:10.617991 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.617998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.618579 (+ 581us) server_negotiation.cc:407] Connection header received
0809 15:06:10.618808 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.618814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.618898 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.618992 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 15:06:10.619588 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.620356 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.622350 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.623154 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.623294 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.623360 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.623632 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.623767 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 15:06:10.624074 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.617781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.129:38541 (local address 127.27.186.131:56193)
0809 15:06:10.618478 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:10.618496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:10.618630 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:10.619032 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:10.619036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:10.619062 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 15:06:10.619432 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.619445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.620528 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.620533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.622168 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:10.622181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.623311 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.623317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:10.623470 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.623492 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 15:06:10.623570 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 15:06:10.623684 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":19,"thread_start_us":905,"threads_started":1}
I20260809 15:06:10.624552 28463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22a18cc1ba1468d8ebcb330c6f8ccf8" candidate_uuid: "2eea1fbf21074c429b56e2c728f9e72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbe493b25124cb6a2722ffa8197f2f8" is_pre_election: true
I20260809 15:06:10.625092 28463 raft_consensus.cc:2454] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eea1fbf21074c429b56e2c728f9e72a in term 0.
I20260809 15:06:10.625531 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22a18cc1ba1468d8ebcb330c6f8ccf8" candidate_uuid: "2eea1fbf21074c429b56e2c728f9e72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" is_pre_election: true
I20260809 15:06:10.625739 28558 leader_election.cc:302] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eea1fbf21074c429b56e2c728f9e72a, edbe493b25124cb6a2722ffa8197f2f8; no voters:
I20260809 15:06:10.625972 28241 raft_consensus.cc:2454] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eea1fbf21074c429b56e2c728f9e72a in term 0.
I20260809 15:06:10.626121 28766 raft_consensus.cc:2781] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:06:10.626259 28766 raft_consensus.cc:482] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:06:10.626360 28766 raft_consensus.cc:3037] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:10.628516 28766 raft_consensus.cc:504] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.629053 28766 leader_election.cc:288] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [CANDIDATE]: Term 1 election: Requested vote from peers f85dd1bfd7424e33a6b5a3042be904ed (127.27.186.129:38541), edbe493b25124cb6a2722ffa8197f2f8 (127.27.186.130:39935)
I20260809 15:06:10.629432 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22a18cc1ba1468d8ebcb330c6f8ccf8" candidate_uuid: "2eea1fbf21074c429b56e2c728f9e72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85dd1bfd7424e33a6b5a3042be904ed"
I20260809 15:06:10.629528 28463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22a18cc1ba1468d8ebcb330c6f8ccf8" candidate_uuid: "2eea1fbf21074c429b56e2c728f9e72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbe493b25124cb6a2722ffa8197f2f8"
I20260809 15:06:10.629635 28241 raft_consensus.cc:3037] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:10.629712 28463 raft_consensus.cc:3037] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:10.632798 28463 raft_consensus.cc:2454] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eea1fbf21074c429b56e2c728f9e72a in term 1.
I20260809 15:06:10.632798 28241 raft_consensus.cc:2454] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eea1fbf21074c429b56e2c728f9e72a in term 1.
I20260809 15:06:10.633172 28558 leader_election.cc:302] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eea1fbf21074c429b56e2c728f9e72a, edbe493b25124cb6a2722ffa8197f2f8; no voters:
I20260809 15:06:10.633425 28766 raft_consensus.cc:2781] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:10.633805 28766 raft_consensus.cc:686] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 1 LEADER]: Becoming Leader. State: Replica: 2eea1fbf21074c429b56e2c728f9e72a, State: Running, Role: LEADER
I20260809 15:06:10.634307 28766 consensus_queue.cc:237] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } }
I20260809 15:06:10.638455 28015 catalog_manager.cc:5495] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eea1fbf21074c429b56e2c728f9e72a (127.27.186.131). New cstate: current_term: 1 leader_uuid: "2eea1fbf21074c429b56e2c728f9e72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eea1fbf21074c429b56e2c728f9e72a" member_type: VOTER last_known_addr { host: "127.27.186.131" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 } health_report { overall_health: UNKNOWN } } }
W20260809 15:06:10.655565 28534 tablet.cc:2369] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:10.688733 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.683761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.27.186.131:34281)
0809 15:06:10.683923 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.683929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.684009 (+ 80us) server_negotiation.cc:407] Connection header received
0809 15:06:10.684205 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.684209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.684290 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.684412 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:10.684779 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.685458 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.686828 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.687229 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.687346 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.687781 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:10.688210 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:10.688357 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.688445 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.688516 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:06:10.696102 28463 raft_consensus.cc:1261] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 1 FOLLOWER]: Refusing update from remote peer 2eea1fbf21074c429b56e2c728f9e72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:10.696102 28241 raft_consensus.cc:1261] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 1 FOLLOWER]: Refusing update from remote peer 2eea1fbf21074c429b56e2c728f9e72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:10.696962 28771 consensus_queue.cc:1035] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "edbe493b25124cb6a2722ffa8197f2f8" member_type: VOTER last_known_addr { host: "127.27.186.130" port: 39935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:10.697312 28766 consensus_queue.cc:1035] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f85dd1bfd7424e33a6b5a3042be904ed" member_type: VOTER last_known_addr { host: "127.27.186.129" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:10.709015 28779 mvcc.cc:205] Tried to move back new op lower bound from 7316635527961980928 to 7316635527722029056. Current Snapshot: MvccSnapshot[applied={T|T < 7316635527961980928}]
I20260809 15:06:10.722322 28016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:10.722543 28016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 15:06:10.725869 28016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a77ef758294403188866da8290b47db: Sending DeleteTablet for 3 replicas of tablet f22a18cc1ba1468d8ebcb330c6f8ccf8
I20260809 15:06:10.726887 28630 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:10 UTC) from {username='slave'} at 127.0.0.1:52146
I20260809 15:06:10.726903 28208 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:10 UTC) from {username='slave'} at 127.0.0.1:50680
I20260809 15:06:10.727311 28431 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 15:06:10 UTC) from {username='slave'} at 127.0.0.1:42656
I20260809 15:06:10.728497 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28107
I20260809 15:06:10.739967 28794 tablet_replica.cc:330] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: stopping tablet replica
I20260809 15:06:10.740350 28794 raft_consensus.cc:2229] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:10.740803 28794 raft_consensus.cc:2258] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:10.741472 28793 tablet_replica.cc:330] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: stopping tablet replica
I20260809 15:06:10.741843 28793 raft_consensus.cc:2229] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:06:10.742292 28793 raft_consensus.cc:2258] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:10.744376 28793 ts_tablet_manager.cc:1905] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:10.747803 28794 ts_tablet_manager.cc:1905] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:10.754469 28792 tablet_replica.cc:330] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: stopping tablet replica
I20260809 15:06:10.754817 28792 raft_consensus.cc:2229] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:06:10.755512 28792 raft_consensus.cc:2258] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:06:10.757678 28792 ts_tablet_manager.cc:1905] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 15:06:10.758445 28794 ts_tablet_manager.cc:1918] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:10.758661 28794 log.cc:1196] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-1/wal/wals/f22a18cc1ba1468d8ebcb330c6f8ccf8
I20260809 15:06:10.759276 28794 ts_tablet_manager.cc:1939] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P edbe493b25124cb6a2722ffa8197f2f8: Deleting consensus metadata
I20260809 15:06:10.759537 28793 ts_tablet_manager.cc:1918] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:10.759686 28793 log.cc:1196] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-0/wal/wals/f22a18cc1ba1468d8ebcb330c6f8ccf8
I20260809 15:06:10.760198 28793 ts_tablet_manager.cc:1939] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P f85dd1bfd7424e33a6b5a3042be904ed: Deleting consensus metadata
I20260809 15:06:10.761343 27987 catalog_manager.cc:4841] TS edbe493b25124cb6a2722ffa8197f2f8 (127.27.186.130:39935): tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 (table test-table [id=43e10425570140618def78bdd15d0177]) successfully deleted
I20260809 15:06:10.762285 27992 catalog_manager.cc:4841] TS f85dd1bfd7424e33a6b5a3042be904ed (127.27.186.129:38541): tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 (table test-table [id=43e10425570140618def78bdd15d0177]) successfully deleted
I20260809 15:06:10.763816 28792 ts_tablet_manager.cc:1918] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 15:06:10.763940 28792 log.cc:1196] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Deleting WAL directory at /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786287660051758-28394-0/minicluster-data/ts-2/wal/wals/f22a18cc1ba1468d8ebcb330c6f8ccf8
I20260809 15:06:10.764392 28792 ts_tablet_manager.cc:1939] T f22a18cc1ba1468d8ebcb330c6f8ccf8 P 2eea1fbf21074c429b56e2c728f9e72a: Deleting consensus metadata
I20260809 15:06:10.765618 27989 catalog_manager.cc:4841] TS 2eea1fbf21074c429b56e2c728f9e72a (127.27.186.131:34281): tablet f22a18cc1ba1468d8ebcb330c6f8ccf8 (table test-table [id=43e10425570140618def78bdd15d0177]) successfully deleted
I20260809 15:06:10.766116 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.729887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.27.186.129:38541)
0809 15:06:10.730099 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.730106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.748540 (+ 18434us) server_negotiation.cc:407] Connection header received
0809 15:06:10.748928 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.748934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.749032 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.749097 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:10.749691 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.750803 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.752558 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.753162 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.753258 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.756133 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:10.756144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:10.756169 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:10.756924 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:10.756987 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:10.757026 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:10.757138 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.765640 (+ 8502us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.765748 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 15:06:10.766762 28306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:10.851876 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28107
I20260809 15:06:10.887827 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28321
I20260809 15:06:10.894655 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:10.888552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.27.186.130:39935)
0809 15:06:10.888758 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:10.888763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:10.888798 (+ 35us) server_negotiation.cc:407] Connection header received
0809 15:06:10.888927 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:10.888933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:10.889041 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:10.889127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:10.889550 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.890365 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.891646 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:10.892045 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:10.892156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:10.893201 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:10.893213 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:10.893227 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:10.893765 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:10.893822 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:10.893843 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:10.893946 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:10.894229 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 15:06:10.894338 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 15:06:10.895239 28521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:11.022997 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28321
I20260809 15:06:11.044345 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28536
I20260809 15:06:11.051301 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:11.045091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.27.186.131:34281)
0809 15:06:11.045291 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:11.045298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:11.045371 (+ 73us) server_negotiation.cc:407] Connection header received
0809 15:06:11.045550 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:11.045555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:11.045647 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:11.045735 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:11.046215 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:11.046994 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:11.048230 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:11.048667 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:11.048785 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:11.049878 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:11.049887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:11.049901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:11.050553 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:11.050599 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:11.050620 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:11.050724 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:11.050878 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 15:06:11.050970 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 15:06:11.051972 28721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:11.167907 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28536
I20260809 15:06:11.187503 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 27965
I20260809 15:06:11.188508 28067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:11.293593 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 27965
2026-08-09T15:06:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2809 ms)
[----------] 6 tests from EncryptionPolicyTest (20878 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:11 dist-test-slave-q9h9 krb5kdc[28808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:11 dist-test-slave-q9h9 krb5kdc[28808](info): set up 2 sockets
Aug 09 15:06:11 dist-test-slave-q9h9 krb5kdc[28808](info): commencing operation
krb5kdc: starting...
W20260809 15:06:12.509645 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:12 dist-test-slave-q9h9 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:12.747087 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:13.054975 28827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:13.055222 28827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:13.055316 28827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:13.061975 28827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:13.062094 28827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:13.062155 28827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:13.062223 28827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:13.062266 28827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:13.071336 28827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:13.072345 28827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:13.073431 28827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:13.084105 28832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:13.084477 28837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:13.084537 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:13.085381 28827 server_base.cc:1034] running on GCE node
I20260809 15:06:13.086076 28827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:13.087249 28827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:13.088435 28827 hybrid_clock.cc:648] HybridClock initialized: now 1786287973088399 us; error 31 us; skew 500 ppm
I20260809 15:06:13.088745 28827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:13 dist-test-slave-q9h9 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:13.091539 28827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:13.092788 28827 webserver.cc:460] Webserver started at http://127.0.0.1:38877/ using document root <none> and password file <none>
I20260809 15:06:13.093369 28827 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:13.093488 28827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:13.093761 28827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:13.096428 28827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "658d4567339e488fa79cdff9d122874d"
format_stamp: "Formatted at 2026-08-09 15:06:13 on dist-test-slave-q9h9"
server_key: "b77da2c7c2e6b9d816e878350456e5b7"
server_key_iv: "dba26a4778b43f3cbff95491ea5ec44f"
server_key_version: "encryptionkey@0"
I20260809 15:06:13.097133 28827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "658d4567339e488fa79cdff9d122874d"
format_stamp: "Formatted at 2026-08-09 15:06:13 on dist-test-slave-q9h9"
server_key: "b77da2c7c2e6b9d816e878350456e5b7"
server_key_iv: "dba26a4778b43f3cbff95491ea5ec44f"
server_key_version: "encryptionkey@0"
I20260809 15:06:13.101691 28827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260809 15:06:13.104782 28852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:13.105945 28827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:06:13.106141 28827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "658d4567339e488fa79cdff9d122874d"
format_stamp: "Formatted at 2026-08-09 15:06:13 on dist-test-slave-q9h9"
server_key: "b77da2c7c2e6b9d816e878350456e5b7"
server_key_iv: "dba26a4778b43f3cbff95491ea5ec44f"
server_key_version: "encryptionkey@0"
I20260809 15:06:13.106308 28827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:13.129154 28827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:13.132701 28827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:13.132939 28827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:13.149873 28827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43405
I20260809 15:06:13.149911 28938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43405 every 8 connection(s)
I20260809 15:06:13.151089 28827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:13.154538 28939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:13.155972 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 28827
I20260809 15:06:13.156196 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:13.157011 28394 external_mini_cluster.cc:1428] Setting key 9d5788ede8cc93f23cc2521f2e7ccf9d
I20260809 15:06:13.168247 28939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d: Bootstrap starting.
Aug 09 15:06:13 dist-test-slave-q9h9 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 15:06:13.171686 28939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:13.172676 28939 log.cc:824] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:13.175352 28939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d: No bootstrap required, opened a new log
I20260809 15:06:13.179052 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:13.158881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34301 (local address 127.0.0.1:43405)
0809 15:06:13.159632 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:13.159642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:13.159692 (+ 50us) server_negotiation.cc:407] Connection header received
0809 15:06:13.161207 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:13.161228 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:13.161613 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:13.161915 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:13.163610 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:13.164872 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:13.166706 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:13.167393 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:13.167510 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:13.172136 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:13.172143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:13.172157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:13.172194 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:13.175113 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:13.175872 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:13.175878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:13.175883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:13.176049 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:13.176490 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:13.176493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:13.176495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:13.176985 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:13.177296 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:13.177794 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 15:06:13.178115 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260809 15:06:13.183038 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:13.183650 28939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658d4567339e488fa79cdff9d122874d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43405 } }
I20260809 15:06:13.183956 28939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:13.184051 28939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658d4567339e488fa79cdff9d122874d, State: Initialized, Role: FOLLOWER
I20260809 15:06:13.184600 28939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658d4567339e488fa79cdff9d122874d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43405 } }
I20260809 15:06:13.184819 28939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:13.184957 28939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:13.185112 28939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:13.187558 28939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658d4567339e488fa79cdff9d122874d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43405 } }
I20260809 15:06:13.188084 28939 leader_election.cc:302] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 658d4567339e488fa79cdff9d122874d; no voters:
I20260809 15:06:13.188664 28939 leader_election.cc:288] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:13.188759 28956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:13.190557 28939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:13.190748 28956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [term 1 LEADER]: Becoming Leader. State: Replica: 658d4567339e488fa79cdff9d122874d, State: Running, Role: LEADER
I20260809 15:06:13.191331 28956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658d4567339e488fa79cdff9d122874d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43405 } }
I20260809 15:06:13.194389 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658d4567339e488fa79cdff9d122874d. Latest consensus state: current_term: 1 leader_uuid: "658d4567339e488fa79cdff9d122874d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658d4567339e488fa79cdff9d122874d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43405 } } }
I20260809 15:06:13.195513 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:13.195200 28958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "658d4567339e488fa79cdff9d122874d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658d4567339e488fa79cdff9d122874d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43405 } } }
I20260809 15:06:13.196054 28958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:13.201578 28969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:13.205129 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:13.187759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50307 (local address 127.0.0.1:43405)
0809 15:06:13.187920 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:13.187926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:13.188290 (+ 364us) server_negotiation.cc:407] Connection header received
0809 15:06:13.188449 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:13.188459 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:13.188535 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:13.188662 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:13.190490 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:13.191728 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:13.193332 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:13.193996 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:13.194141 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:13.197443 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:13.197450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:13.197455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:13.197493 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:13.201178 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:13.201986 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:13.201992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:13.201995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:13.202138 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:13.202651 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:13.202656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:13.202659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:13.203056 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:13.203328 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:13.204527 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0809 15:06:13.204723 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:06:13.206465 28969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:13.216531 28969 catalog_manager.cc:1347] Generated new cluster ID: 1c9323efcec3414b9ecf6007aae68a33
I20260809 15:06:13.216629 28969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:13.223227 28969 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:13.224810 28969 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:13.233364 28969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 658d4567339e488fa79cdff9d122874d: Generated new TSK 0
I20260809 15:06:13.234225 28969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:13.239784 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 28827
I20260809 15:06:13.241321 28932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:13.325807 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 28827
W20260809 15:06:13.345111 28806 connection.cc:535] client connection to 127.0.0.1:43405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43405): Connection reset by peer (error 104)
2026-08-09T15:06:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:13 dist-test-slave-q9h9 krb5kdc[28994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:13 dist-test-slave-q9h9 krb5kdc[28994](info): set up 2 sockets
Aug 09 15:06:13 dist-test-slave-q9h9 krb5kdc[28994](info): commencing operation
krb5kdc: starting...
W20260809 15:06:14.527951 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:14 dist-test-slave-q9h9 krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:14.764642 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:15.070462 29009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:15.070779 29009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:15.070904 29009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:15.077728 29009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:15.077829 29009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:15.077903 29009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:15.077989 29009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:15.078063 29009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:15.087136 29009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:15.088184 29009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:15.089354 29009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:15.099821 29024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:15.099881 29022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:15.099855 29021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:15.100711 29009 server_base.cc:1034] running on GCE node
I20260809 15:06:15.101385 29009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:15.102615 29009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:15.103840 29009 hybrid_clock.cc:648] HybridClock initialized: now 1786287975103788 us; error 38 us; skew 500 ppm
I20260809 15:06:15.104166 29009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:15 dist-test-slave-q9h9 krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:15.107029 29009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:15.108362 29009 webserver.cc:460] Webserver started at http://127.0.0.1:36949/ using document root <none> and password file <none>
I20260809 15:06:15.108932 29009 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:15.109046 29009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:15.109321 29009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:15.112057 29009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "a9d38316e76d4780ac04cadc8b6b0982"
format_stamp: "Formatted at 2026-08-09 15:06:15 on dist-test-slave-q9h9"
server_key: "db7289e177444b6c3bedbf5bfa1e4852"
server_key_iv: "e7ea9dbc73ef7e41c0f95a6f35b03b73"
server_key_version: "encryptionkey@0"
I20260809 15:06:15.112782 29009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "a9d38316e76d4780ac04cadc8b6b0982"
format_stamp: "Formatted at 2026-08-09 15:06:15 on dist-test-slave-q9h9"
server_key: "db7289e177444b6c3bedbf5bfa1e4852"
server_key_iv: "e7ea9dbc73ef7e41c0f95a6f35b03b73"
server_key_version: "encryptionkey@0"
I20260809 15:06:15.117399 29009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 15:06:15.120498 29032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:15.121649 29009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 15:06:15.121794 29009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "a9d38316e76d4780ac04cadc8b6b0982"
format_stamp: "Formatted at 2026-08-09 15:06:15 on dist-test-slave-q9h9"
server_key: "db7289e177444b6c3bedbf5bfa1e4852"
server_key_iv: "e7ea9dbc73ef7e41c0f95a6f35b03b73"
server_key_version: "encryptionkey@0"
I20260809 15:06:15.121958 29009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:15.155395 29009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:15.158825 29009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:15.159060 29009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:15.175369 29009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46269
I20260809 15:06:15.175398 29112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46269 every 8 connection(s)
I20260809 15:06:15.176568 29009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:15.179881 29113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:15.183547 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29009
I20260809 15:06:15.183694 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:15.184094 28394 external_mini_cluster.cc:1428] Setting key f158a3cb5d6e614611c79571d0346278
I20260809 15:06:15.192408 29113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982: Bootstrap starting.
I20260809 15:06:15.196051 29113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:15.197314 29113 log.cc:824] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:15.197557 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:15.185653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47231 (local address 127.0.0.1:46269)
0809 15:06:15.186465 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:15.186479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:15.186527 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:06:15.188163 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:15.188190 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:15.188675 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:15.189085 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:15.189556 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:15.191512 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:15.192803 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:15.193768 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:15.193984 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:15.195074 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:15.195085 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:15.195102 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:15.195736 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:15.195835 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:15.195867 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:15.196031 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:15.196204 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 15:06:15.196359 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":357,"threads_started":1}
I20260809 15:06:15.200039 29113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982: No bootstrap required, opened a new log
I20260809 15:06:15.201159 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:15.209422 29113 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d38316e76d4780ac04cadc8b6b0982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260809 15:06:15.209733 29113 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:15.209841 29113 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d38316e76d4780ac04cadc8b6b0982, State: Initialized, Role: FOLLOWER
I20260809 15:06:15.210512 29113 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d38316e76d4780ac04cadc8b6b0982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260809 15:06:15.210809 29113 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:15.210934 29113 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:15.211083 29113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:15.212337 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:15.206230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54601 (local address 127.0.0.1:46269)
0809 15:06:15.206386 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:15.206392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:15.206718 (+ 326us) server_negotiation.cc:407] Connection header received
0809 15:06:15.206860 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:15.206865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:15.206928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:15.207014 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:15.207445 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:15.208556 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:15.209815 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:15.210464 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:15.210593 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:15.211616 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:15.211623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:15.211628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:15.211651 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:15.211696 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:15.211715 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:15.211856 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:15.211971 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 15:06:15.212022 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:06:15.213680 29113 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d38316e76d4780ac04cadc8b6b0982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260809 15:06:15.214157 29113 leader_election.cc:302] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d38316e76d4780ac04cadc8b6b0982; no voters:
I20260809 15:06:15.214663 29113 leader_election.cc:288] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:15.214777 29127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:15.215833 29127 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [term 1 LEADER]: Becoming Leader. State: Replica: a9d38316e76d4780ac04cadc8b6b0982, State: Running, Role: LEADER
I20260809 15:06:15.216348 29127 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d38316e76d4780ac04cadc8b6b0982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260809 15:06:15.216581 29113 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:15.218863 29131 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d38316e76d4780ac04cadc8b6b0982. Latest consensus state: current_term: 1 leader_uuid: "a9d38316e76d4780ac04cadc8b6b0982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d38316e76d4780ac04cadc8b6b0982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } } }
I20260809 15:06:15.219218 29131 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:15.219787 29128 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9d38316e76d4780ac04cadc8b6b0982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d38316e76d4780ac04cadc8b6b0982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } } }
I20260809 15:06:15.220083 29128 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:15.222899 29140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:15.227845 29140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:15.236467 29140 catalog_manager.cc:1347] Generated new cluster ID: 708b01afb3f84308803b5502e3e29ff9
I20260809 15:06:15.236585 29140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:15.259950 29140 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:15.261847 29140 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:15.270197 29140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9d38316e76d4780ac04cadc8b6b0982: Generated new TSK 0
I20260809 15:06:15.271121 29140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:15.278599 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29009
W20260809 15:06:15.279728 29104 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47231
W20260809 15:06:15.280398 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:15.280548 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29009
W20260809 15:06:15.301726 28992 connection.cc:535] client connection to 127.0.0.1:46269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46269): Connection reset by peer (error 104)
2026-08-09T15:06:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:15 dist-test-slave-q9h9 krb5kdc[29161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:15 dist-test-slave-q9h9 krb5kdc[29161](info): set up 2 sockets
Aug 09 15:06:15 dist-test-slave-q9h9 krb5kdc[29161](info): commencing operation
krb5kdc: starting...
W20260809 15:06:16.507889 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:16 dist-test-slave-q9h9 krb5kdc[29161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:16.754817 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:17.074671 29185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:17.074963 29185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:17.075115 29185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:17.082086 29185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:17.082190 29185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:17.082275 29185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:17.082369 29185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:17.082433 29185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:17.091827 29185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:17.092918 29185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:17.094138 29185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:17.104877 29192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:17.104907 29194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:17.104998 29191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:17.105544 29185 server_base.cc:1034] running on GCE node
I20260809 15:06:17.106247 29185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:17.107513 29185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:17.108698 29185 hybrid_clock.cc:648] HybridClock initialized: now 1786287977108654 us; error 42 us; skew 500 ppm
I20260809 15:06:17.109002 29185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:17 dist-test-slave-q9h9 krb5kdc[29161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:17.114995 29185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:17.116266 29185 webserver.cc:460] Webserver started at http://127.0.0.1:44455/ using document root <none> and password file <none>
I20260809 15:06:17.116873 29185 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:17.116992 29185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:17.117256 29185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:17.120050 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "893b1b634f4a42a1a8394c4f031584c0"
format_stamp: "Formatted at 2026-08-09 15:06:17 on dist-test-slave-q9h9"
server_key: "b2cbd12278409ce1c58a009610f2f5bb"
server_key_iv: "485a437eb2009eb6a5963d3d3a4208c7"
server_key_version: "encryptionkey@0"
I20260809 15:06:17.120759 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "893b1b634f4a42a1a8394c4f031584c0"
format_stamp: "Formatted at 2026-08-09 15:06:17 on dist-test-slave-q9h9"
server_key: "b2cbd12278409ce1c58a009610f2f5bb"
server_key_iv: "485a437eb2009eb6a5963d3d3a4208c7"
server_key_version: "encryptionkey@0"
I20260809 15:06:17.125401 29185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 15:06:17.128582 29203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:17.129741 29185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:06:17.129901 29185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "893b1b634f4a42a1a8394c4f031584c0"
format_stamp: "Formatted at 2026-08-09 15:06:17 on dist-test-slave-q9h9"
server_key: "b2cbd12278409ce1c58a009610f2f5bb"
server_key_iv: "485a437eb2009eb6a5963d3d3a4208c7"
server_key_version: "encryptionkey@0"
I20260809 15:06:17.130118 29185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:17.135176 29185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:17.139014 29185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:17.139268 29185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:17.154474 29185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34885
I20260809 15:06:17.154527 29288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34885 every 8 connection(s)
I20260809 15:06:17.155687 29185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:17.158237 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29185
I20260809 15:06:17.158396 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:17.158784 28394 external_mini_cluster.cc:1428] Setting key 98e1fb08526ab6cbefa02abc3ad8df91
I20260809 15:06:17.159153 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:17.166196 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:17.160292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42785 (local address 127.0.0.1:34885)
0809 15:06:17.161017 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:17.161032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:17.161079 (+ 47us) server_negotiation.cc:407] Connection header received
0809 15:06:17.162650 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:17.162676 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:17.163082 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:17.163486 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:17.164391 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:17.164400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:17.164415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:17.164431 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:17.164525 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:17.164551 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:17.164755 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:17.164877 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 15:06:17.165055 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
I20260809 15:06:17.169426 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:17.173269 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0: Bootstrap starting.
I20260809 15:06:17.176796 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:17.174595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56167 (local address 127.0.0.1:34885)
0809 15:06:17.174769 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:17.174774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:17.175165 (+ 391us) server_negotiation.cc:407] Connection header received
0809 15:06:17.175298 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:17.175302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:17.175353 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:17.175447 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:17.176194 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:17.176199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:17.176204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:17.176219 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:17.176259 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:17.176288 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:17.176397 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:17.176470 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 15:06:17.176518 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:06:17.176764 29289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:17.177997 29289 log.cc:824] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:17.180358 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0: No bootstrap required, opened a new log
I20260809 15:06:17.187340 29289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893b1b634f4a42a1a8394c4f031584c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260809 15:06:17.187638 29289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:17.187706 29289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893b1b634f4a42a1a8394c4f031584c0, State: Initialized, Role: FOLLOWER
I20260809 15:06:17.188140 29289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893b1b634f4a42a1a8394c4f031584c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260809 15:06:17.188324 29289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:17.188416 29289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:17.188549 29289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:17.190699 29289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893b1b634f4a42a1a8394c4f031584c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260809 15:06:17.191114 29289 leader_election.cc:302] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893b1b634f4a42a1a8394c4f031584c0; no voters:
I20260809 15:06:17.191581 29289 leader_election.cc:288] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:17.191711 29304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:17.192341 29304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [term 1 LEADER]: Becoming Leader. State: Replica: 893b1b634f4a42a1a8394c4f031584c0, State: Running, Role: LEADER
I20260809 15:06:17.192837 29304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893b1b634f4a42a1a8394c4f031584c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260809 15:06:17.193320 29289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:17.195206 29307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893b1b634f4a42a1a8394c4f031584c0. Latest consensus state: current_term: 1 leader_uuid: "893b1b634f4a42a1a8394c4f031584c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893b1b634f4a42a1a8394c4f031584c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } } }
I20260809 15:06:17.195631 29307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:17.196081 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "893b1b634f4a42a1a8394c4f031584c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893b1b634f4a42a1a8394c4f031584c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } } }
I20260809 15:06:17.196776 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:17.202625 29314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:17.208448 29314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:17.215734 29314 catalog_manager.cc:1347] Generated new cluster ID: 3d1e19bfa45449e08aaeafd04306dca6
I20260809 15:06:17.215833 29314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:17.236265 29314 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:17.236702 29314 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:17.243381 29314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 893b1b634f4a42a1a8394c4f031584c0: Generated new TSK 0
I20260809 15:06:17.244215 29314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:17.248085 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29185
W20260809 15:06:17.249049 29279 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42785
W20260809 15:06:17.249716 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:17.249843 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29185
2026-08-09T15:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:17 dist-test-slave-q9h9 krb5kdc[29337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:17 dist-test-slave-q9h9 krb5kdc[29337](info): set up 2 sockets
Aug 09 15:06:17 dist-test-slave-q9h9 krb5kdc[29337](info): commencing operation
krb5kdc: starting...
W20260809 15:06:18.493376 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:18 dist-test-slave-q9h9 krb5kdc[29337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:18.734174 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:19.043746 29358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:19.044006 29358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:19.044147 29358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:19.050910 29358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:19.051017 29358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:19.051093 29358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:19.051179 29358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:19.051240 29358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:19.060338 29358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:19.061371 29358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:19.062566 29358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:19.073164 29371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:19.073227 29365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:19.073258 29366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:19.073818 29358 server_base.cc:1034] running on GCE node
I20260809 15:06:19.074580 29358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:19.075817 29358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:19.077013 29358 hybrid_clock.cc:648] HybridClock initialized: now 1786287979076977 us; error 27 us; skew 500 ppm
I20260809 15:06:19.077327 29358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:19 dist-test-slave-q9h9 krb5kdc[29337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:19.080268 29358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:19.081501 29358 webserver.cc:460] Webserver started at http://127.0.0.1:42451/ using document root <none> and password file <none>
I20260809 15:06:19.082093 29358 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:19.082216 29358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:19.082475 29358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:19.085034 29358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "82e28ffd427d4104bc1ac27fb52f0ba5"
format_stamp: "Formatted at 2026-08-09 15:06:19 on dist-test-slave-q9h9"
server_key: "b30eb8f064c977f67416e0de9c75df0d"
server_key_iv: "c8e1fe924b203aa1700857f8334882e0"
server_key_version: "encryptionkey@0"
I20260809 15:06:19.085709 29358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "82e28ffd427d4104bc1ac27fb52f0ba5"
format_stamp: "Formatted at 2026-08-09 15:06:19 on dist-test-slave-q9h9"
server_key: "b30eb8f064c977f67416e0de9c75df0d"
server_key_iv: "c8e1fe924b203aa1700857f8334882e0"
server_key_version: "encryptionkey@0"
I20260809 15:06:19.090286 29358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 15:06:19.093367 29382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:19.094569 29358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:19.094720 29358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "82e28ffd427d4104bc1ac27fb52f0ba5"
format_stamp: "Formatted at 2026-08-09 15:06:19 on dist-test-slave-q9h9"
server_key: "b30eb8f064c977f67416e0de9c75df0d"
server_key_iv: "c8e1fe924b203aa1700857f8334882e0"
server_key_version: "encryptionkey@0"
I20260809 15:06:19.094890 29358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:19.131909 29358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:19.135753 29358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:19.135991 29358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:19.152549 29358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34967
I20260809 15:06:19.152567 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34967 every 8 connection(s)
I20260809 15:06:19.153846 29358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:19.157501 29462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:19.163483 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29358
I20260809 15:06:19.163642 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:19.164065 28394 external_mini_cluster.cc:1428] Setting key 992492da4ee35ddc5e3ccaf4b65ff527
I20260809 15:06:19.167543 29462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5: Bootstrap starting.
I20260809 15:06:19.171010 29462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:19.172042 29462 log.cc:824] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:19.174814 29462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5: No bootstrap required, opened a new log
Aug 09 15:06:19 dist-test-slave-q9h9 krb5kdc[29337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 15:06:19.182721 29462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260809 15:06:19.183019 29462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:19.183120 29462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e28ffd427d4104bc1ac27fb52f0ba5, State: Initialized, Role: FOLLOWER
I20260809 15:06:19.183689 29462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260809 15:06:19.183924 29462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:19.184058 29462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:19.184279 29462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:19.186101 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:19.165679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54775 (local address 127.0.0.1:34967)
0809 15:06:19.166426 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:19.166437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:19.166474 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:06:19.168091 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:19.168111 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:19.168459 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:19.168740 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:19.170617 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:19.171920 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:19.173367 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:19.174000 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:19.174153 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:19.178825 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:19.178833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:19.178846 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:19.178878 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:19.181838 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:19.182542 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:19.182548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:19.182554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:19.182714 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:19.183094 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:19.183097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:19.183099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:19.183618 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:19.183907 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:19.184354 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 15:06:19.184783 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":356,"threads_started":1}
I20260809 15:06:19.187214 29462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260809 15:06:19.187755 29462 leader_election.cc:302] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e28ffd427d4104bc1ac27fb52f0ba5; no voters:
I20260809 15:06:19.188436 29462 leader_election.cc:288] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:19.188580 29470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:19.189268 29470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [term 1 LEADER]: Becoming Leader. State: Replica: 82e28ffd427d4104bc1ac27fb52f0ba5, State: Running, Role: LEADER
I20260809 15:06:19.189805 29470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260809 15:06:19.190384 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:19.192312 29474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } } }
I20260809 15:06:19.192775 29474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:19.194690 29470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e28ffd427d4104bc1ac27fb52f0ba5. Latest consensus state: current_term: 1 leader_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e28ffd427d4104bc1ac27fb52f0ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } } }
I20260809 15:06:19.195281 29470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:19.203595 29462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:19.203770 29476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:19.208678 29476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:19.221905 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:19.204257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43095 (local address 127.0.0.1:34967)
0809 15:06:19.204414 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:19.204419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:19.204952 (+ 533us) server_negotiation.cc:407] Connection header received
0809 15:06:19.205120 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:19.205125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:19.205196 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:19.205299 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:19.207209 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:19.208436 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:19.210342 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:19.211110 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:19.211237 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:19.214281 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:19.214289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:19.214294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:19.214336 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:19.218052 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:19.218940 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:19.218946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:19.218950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:19.219100 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:19.219530 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:19.219535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:19.219539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:19.219966 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:19.220198 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:19.221283 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0809 15:06:19.221520 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 15:06:19.232709 29476 catalog_manager.cc:1347] Generated new cluster ID: b48af6a912e54f338ac16afa906cadae
I20260809 15:06:19.232813 29476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:19.255944 29476 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:19.257236 29476 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:19.269973 29476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e28ffd427d4104bc1ac27fb52f0ba5: Generated new TSK 0
I20260809 15:06:19.270797 29476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:19.278855 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29358
I20260809 15:06:19.279875 29452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:19.369565 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29358
2026-08-09T15:06:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2114 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:19 dist-test-slave-q9h9 krb5kdc[29521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:19 dist-test-slave-q9h9 krb5kdc[29521](info): set up 2 sockets
Aug 09 15:06:19 dist-test-slave-q9h9 krb5kdc[29521](info): commencing operation
krb5kdc: starting...
W20260809 15:06:20.577936 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:20 dist-test-slave-q9h9 krb5kdc[29521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:20.824863 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:21.133008 29543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:21.133260 29543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:21.133386 29543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:21.140218 29543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:21.140316 29543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:21.140390 29543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:21.140473 29543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:21.140530 29543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:21.149602 29543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:21.150681 29543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:21.151840 29543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:21.162338 29551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:21.162395 29553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:21.162665 29549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:21.163111 29543 server_base.cc:1034] running on GCE node
I20260809 15:06:21.163815 29543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:21.165005 29543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:21.166194 29543 hybrid_clock.cc:648] HybridClock initialized: now 1786287981166153 us; error 34 us; skew 500 ppm
I20260809 15:06:21.166501 29543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:21 dist-test-slave-q9h9 krb5kdc[29521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:21.169380 29543 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:21.170676 29543 webserver.cc:460] Webserver started at http://127.0.0.1:32845/ using document root <none> and password file <none>
I20260809 15:06:21.171257 29543 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:21.171372 29543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:21.171635 29543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:21.174417 29543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "3396a5e0c325497bbb1285b90b2d5389"
format_stamp: "Formatted at 2026-08-09 15:06:21 on dist-test-slave-q9h9"
server_key: "5e6dcafa4948d786032ca60fb940f0df"
server_key_iv: "5120779bdae03ca8c387d60e54bf6b3e"
server_key_version: "encryptionkey@0"
I20260809 15:06:21.175125 29543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "3396a5e0c325497bbb1285b90b2d5389"
format_stamp: "Formatted at 2026-08-09 15:06:21 on dist-test-slave-q9h9"
server_key: "5e6dcafa4948d786032ca60fb940f0df"
server_key_iv: "5120779bdae03ca8c387d60e54bf6b3e"
server_key_version: "encryptionkey@0"
I20260809 15:06:21.179698 29543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:06:21.182924 29566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:21.184079 29543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:21.184221 29543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "3396a5e0c325497bbb1285b90b2d5389"
format_stamp: "Formatted at 2026-08-09 15:06:21 on dist-test-slave-q9h9"
server_key: "5e6dcafa4948d786032ca60fb940f0df"
server_key_iv: "5120779bdae03ca8c387d60e54bf6b3e"
server_key_version: "encryptionkey@0"
I20260809 15:06:21.184377 29543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:21.195945 29543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:21.199753 29543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:21.199990 29543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:21.216331 29543 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36551
I20260809 15:06:21.216368 29655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36551 every 8 connection(s)
I20260809 15:06:21.217556 29543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:21.220918 29656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:21.224714 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29543
I20260809 15:06:21.224890 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:21.225432 28394 external_mini_cluster.cc:1428] Setting key 7447e0d06362fdac29068c25936adaf5
I20260809 15:06:21.233290 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389: Bootstrap starting.
I20260809 15:06:21.236696 29656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:21.237805 29656 log.cc:824] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:21.237838 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:21.227175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46761 (local address 127.0.0.1:36551)
0809 15:06:21.227917 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:21.227928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:21.227967 (+ 39us) server_negotiation.cc:407] Connection header received
0809 15:06:21.229539 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:21.229559 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:21.229930 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:21.230291 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:21.230833 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:21.232087 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:21.233835 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:21.234520 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:21.234653 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:21.235819 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:21.235826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:21.235838 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:21.236330 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:21.236398 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:21.236422 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:21.236503 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:21.236736 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 15:06:21.236946 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":353,"threads_started":1}
I20260809 15:06:21.240485 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389: No bootstrap required, opened a new log
I20260809 15:06:21.241328 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:21.248695 29656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3396a5e0c325497bbb1285b90b2d5389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260809 15:06:21.249154 29656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:21.249255 29656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3396a5e0c325497bbb1285b90b2d5389, State: Initialized, Role: FOLLOWER
I20260809 15:06:21.249851 29656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3396a5e0c325497bbb1285b90b2d5389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260809 15:06:21.250111 29656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:21.250272 29656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:21.250459 29656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:21.251942 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:21.245410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40931 (local address 127.0.0.1:36551)
0809 15:06:21.245585 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:21.245590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:21.245955 (+ 365us) server_negotiation.cc:407] Connection header received
0809 15:06:21.246142 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:21.246146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:21.246208 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:21.246297 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:21.246890 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:21.247612 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:21.249539 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:21.250068 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:21.250192 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:21.251297 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:21.251302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:21.251305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:21.251321 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:21.251363 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:21.251381 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:21.251460 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:21.251584 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 15:06:21.251652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 15:06:21.253198 29656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3396a5e0c325497bbb1285b90b2d5389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260809 15:06:21.253667 29656 leader_election.cc:302] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3396a5e0c325497bbb1285b90b2d5389; no voters:
I20260809 15:06:21.254218 29656 leader_election.cc:288] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:21.254364 29671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:21.255003 29671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [term 1 LEADER]: Becoming Leader. State: Replica: 3396a5e0c325497bbb1285b90b2d5389, State: Running, Role: LEADER
I20260809 15:06:21.255642 29671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3396a5e0c325497bbb1285b90b2d5389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260809 15:06:21.258244 29674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3396a5e0c325497bbb1285b90b2d5389. Latest consensus state: current_term: 1 leader_uuid: "3396a5e0c325497bbb1285b90b2d5389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3396a5e0c325497bbb1285b90b2d5389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } } }
I20260809 15:06:21.258525 29675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3396a5e0c325497bbb1285b90b2d5389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3396a5e0c325497bbb1285b90b2d5389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } } }
I20260809 15:06:21.258728 29674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:21.258873 29675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:21.259907 29656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:21.261633 29681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:21.267326 29681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:21.275915 29681 catalog_manager.cc:1347] Generated new cluster ID: c6d30e981ff24d2db199006698dd36d3
I20260809 15:06:21.276013 29681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:21.291821 29681 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:21.293529 29681 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:21.302264 29681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3396a5e0c325497bbb1285b90b2d5389: Generated new TSK 0
I20260809 15:06:21.303160 29681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:21.312856 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29543
W20260809 15:06:21.313781 29647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46761
W20260809 15:06:21.314381 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29543: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:21.314505 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29543
2026-08-09T15:06:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:21 dist-test-slave-q9h9 krb5kdc[29704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:21 dist-test-slave-q9h9 krb5kdc[29704](info): set up 2 sockets
Aug 09 15:06:21 dist-test-slave-q9h9 krb5kdc[29704](info): commencing operation
krb5kdc: starting...
W20260809 15:06:22.539445 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:22 dist-test-slave-q9h9 krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:22.788285 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:23.095930 29725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:23.096182 29725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:23.096307 29725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:23.103140 29725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:23.103240 29725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:23.103313 29725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:23.103368 29725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:23.103403 29725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:23.112577 29725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:23.113692 29725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:23.114934 29725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:23.125866 29731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:23.125811 29733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:23.125797 29736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:23.126627 29725 server_base.cc:1034] running on GCE node
I20260809 15:06:23.127329 29725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:23.128561 29725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:23.129770 29725 hybrid_clock.cc:648] HybridClock initialized: now 1786287983129713 us; error 50 us; skew 500 ppm
I20260809 15:06:23.130133 29725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:23 dist-test-slave-q9h9 krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:23.137573 29725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:23.138911 29725 webserver.cc:460] Webserver started at http://127.0.0.1:39503/ using document root <none> and password file <none>
I20260809 15:06:23.139521 29725 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:23.139638 29725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:23.139918 29725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:23.142704 29725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "f9f61f742ede4549885c1171af0b5a1a"
format_stamp: "Formatted at 2026-08-09 15:06:23 on dist-test-slave-q9h9"
server_key: "81a48e617bdaff36cbdaa3276ffdb6e6"
server_key_iv: "dc384392539f0444309496c88c274651"
server_key_version: "encryptionkey@0"
I20260809 15:06:23.143429 29725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "f9f61f742ede4549885c1171af0b5a1a"
format_stamp: "Formatted at 2026-08-09 15:06:23 on dist-test-slave-q9h9"
server_key: "81a48e617bdaff36cbdaa3276ffdb6e6"
server_key_iv: "dc384392539f0444309496c88c274651"
server_key_version: "encryptionkey@0"
I20260809 15:06:23.148136 29725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:06:23.151271 29749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:23.152446 29725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:23.152604 29725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "f9f61f742ede4549885c1171af0b5a1a"
format_stamp: "Formatted at 2026-08-09 15:06:23 on dist-test-slave-q9h9"
server_key: "81a48e617bdaff36cbdaa3276ffdb6e6"
server_key_iv: "dc384392539f0444309496c88c274651"
server_key_version: "encryptionkey@0"
I20260809 15:06:23.152783 29725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:23.158073 29725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:23.161484 29725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:23.161743 29725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:23.178001 29725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43733
I20260809 15:06:23.178100 29842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43733 every 8 connection(s)
I20260809 15:06:23.179203 29725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:23.182510 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:23.188421 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29725
I20260809 15:06:23.188591 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:23.188997 28394 external_mini_cluster.cc:1428] Setting key ab8ea44b51f0d51ce1f0890d45d79ccc
I20260809 15:06:23.192605 29844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a: Bootstrap starting.
I20260809 15:06:23.195887 29844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:23.196092 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:23.190551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38495 (local address 127.0.0.1:43733)
0809 15:06:23.191383 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:23.191394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:23.191433 (+ 39us) server_negotiation.cc:407] Connection header received
0809 15:06:23.193068 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:23.193090 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:23.193378 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:23.193653 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:23.194698 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:23.194704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:23.194716 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:23.194731 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:23.194787 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:23.194805 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:23.194966 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:23.195083 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 15:06:23.195251 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":430,"threads_started":1}
I20260809 15:06:23.196906 29844 log.cc:824] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:23.199503 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:23.199702 29844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a: No bootstrap required, opened a new log
I20260809 15:06:23.206667 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:23.204468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59375 (local address 127.0.0.1:43733)
0809 15:06:23.204637 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:23.204642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:23.204978 (+ 336us) server_negotiation.cc:407] Connection header received
0809 15:06:23.205112 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:23.205115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:23.205155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:23.205249 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:23.206086 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:23.206089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:23.206092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:23.206105 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:23.206137 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:23.206154 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:23.206255 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:23.206367 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 15:06:23.206411 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 15:06:23.208889 29844 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f61f742ede4549885c1171af0b5a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260809 15:06:23.209190 29844 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:23.209281 29844 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f61f742ede4549885c1171af0b5a1a, State: Initialized, Role: FOLLOWER
I20260809 15:06:23.209796 29844 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f61f742ede4549885c1171af0b5a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260809 15:06:23.210057 29844 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:23.210186 29844 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:23.210343 29844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:23.212361 29844 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f61f742ede4549885c1171af0b5a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260809 15:06:23.212891 29844 leader_election.cc:302] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9f61f742ede4549885c1171af0b5a1a; no voters:
I20260809 15:06:23.213389 29844 leader_election.cc:288] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:23.213506 29861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:23.214176 29861 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [term 1 LEADER]: Becoming Leader. State: Replica: f9f61f742ede4549885c1171af0b5a1a, State: Running, Role: LEADER
I20260809 15:06:23.214653 29861 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f61f742ede4549885c1171af0b5a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260809 15:06:23.215246 29844 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:23.217234 29862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9f61f742ede4549885c1171af0b5a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f61f742ede4549885c1171af0b5a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } } }
I20260809 15:06:23.217209 29864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9f61f742ede4549885c1171af0b5a1a. Latest consensus state: current_term: 1 leader_uuid: "f9f61f742ede4549885c1171af0b5a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f61f742ede4549885c1171af0b5a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } } }
I20260809 15:06:23.217661 29862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:23.217814 29864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:23.224953 29868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:23.229912 29868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:23.238444 29868 catalog_manager.cc:1347] Generated new cluster ID: 4ec91eb1971e47edae81010bd4effc97
I20260809 15:06:23.238538 29868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:23.258792 29868 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:23.259204 29868 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:23.270296 29868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9f61f742ede4549885c1171af0b5a1a: Generated new TSK 0
I20260809 15:06:23.271064 29868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:23.278990 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29725
W20260809 15:06:23.279889 29827 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38495
W20260809 15:06:23.280509 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:23.280624 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29725
2026-08-09T15:06:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:23 dist-test-slave-q9h9 krb5kdc[29898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:23 dist-test-slave-q9h9 krb5kdc[29898](info): set up 2 sockets
Aug 09 15:06:23 dist-test-slave-q9h9 krb5kdc[29898](info): commencing operation
krb5kdc: starting...
W20260809 15:06:24.493125 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:24 dist-test-slave-q9h9 krb5kdc[29898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:24.738762 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:25.052297 29917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:25.052551 29917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:25.052692 29917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:25.059563 29917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:25.059667 29917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:25.059759 29917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:25.059846 29917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:25.059906 29917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:25.068991 29917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:25.070034 29917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:25.071194 29917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:25.081705 29924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:25.081738 29928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:25.082134 29933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:25.082579 29917 server_base.cc:1034] running on GCE node
I20260809 15:06:25.083236 29917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:25.084430 29917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:25.085625 29917 hybrid_clock.cc:648] HybridClock initialized: now 1786287985085574 us; error 41 us; skew 500 ppm
I20260809 15:06:25.085929 29917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:25 dist-test-slave-q9h9 krb5kdc[29898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:25.088855 29917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:25.090215 29917 webserver.cc:460] Webserver started at http://127.0.0.1:44571/ using document root <none> and password file <none>
I20260809 15:06:25.090838 29917 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:25.090961 29917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:25.091224 29917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:25.093871 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "9422c1f3e2b94f4c8b3227453de89238"
format_stamp: "Formatted at 2026-08-09 15:06:25 on dist-test-slave-q9h9"
server_key: "f6998693b0ccf281f31b6b8023cecad7"
server_key_iv: "ad7eba7e89315243d7155f50b1733aa4"
server_key_version: "encryptionkey@0"
I20260809 15:06:25.094605 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "9422c1f3e2b94f4c8b3227453de89238"
format_stamp: "Formatted at 2026-08-09 15:06:25 on dist-test-slave-q9h9"
server_key: "f6998693b0ccf281f31b6b8023cecad7"
server_key_iv: "ad7eba7e89315243d7155f50b1733aa4"
server_key_version: "encryptionkey@0"
I20260809 15:06:25.099186 29917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 15:06:25.102401 29942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:25.103525 29917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 15:06:25.103686 29917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "9422c1f3e2b94f4c8b3227453de89238"
format_stamp: "Formatted at 2026-08-09 15:06:25 on dist-test-slave-q9h9"
server_key: "f6998693b0ccf281f31b6b8023cecad7"
server_key_iv: "ad7eba7e89315243d7155f50b1733aa4"
server_key_version: "encryptionkey@0"
I20260809 15:06:25.103861 29917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:25.115166 29917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:25.118467 29917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:25.118697 29917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:25.135272 29917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41411
I20260809 15:06:25.135309 30029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41411 every 8 connection(s)
I20260809 15:06:25.136457 29917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:25.139758 30030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:25.141783 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 29917
I20260809 15:06:25.141949 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:25.142380 28394 external_mini_cluster.cc:1428] Setting key dcb3acb99ae6d8abd93141aa09e4e0fd
I20260809 15:06:25.150820 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238: Bootstrap starting.
I20260809 15:06:25.154278 30030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:25.155550 30030 log.cc:824] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238: Log is configured to *not* fsync() on all Append() calls
Aug 09 15:06:25 dist-test-slave-q9h9 krb5kdc[29898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 15:06:25.158800 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238: No bootstrap required, opened a new log
I20260809 15:06:25.164556 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:25.144030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:56019 (local address 127.0.0.1:41411)
0809 15:06:25.144862 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:25.144872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:25.144913 (+ 41us) server_negotiation.cc:407] Connection header received
0809 15:06:25.146568 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:25.146589 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:25.146945 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:25.147246 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:25.149046 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:25.150334 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:25.152042 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:25.152732 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:25.152865 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:25.157510 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:25.157517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:25.157531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:25.157568 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:25.160489 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:25.161423 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:25.161429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:25.161434 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:25.161597 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:25.162088 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:25.162091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:25.162093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:25.162572 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:25.162802 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:25.163314 (+ 512us) server_negotiation.cc:299] Negotiation successful
0809 15:06:25.163582 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":354,"threads_started":1}
I20260809 15:06:25.167027 30030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9422c1f3e2b94f4c8b3227453de89238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41411 } }
I20260809 15:06:25.167416 30030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:25.167510 30030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9422c1f3e2b94f4c8b3227453de89238, State: Initialized, Role: FOLLOWER
I20260809 15:06:25.168198 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:25.168114 30030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9422c1f3e2b94f4c8b3227453de89238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41411 } }
I20260809 15:06:25.168340 30030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:25.168502 30030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:25.168650 30030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:25.170749 30030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9422c1f3e2b94f4c8b3227453de89238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41411 } }
I20260809 15:06:25.171286 30030 leader_election.cc:302] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9422c1f3e2b94f4c8b3227453de89238; no voters:
I20260809 15:06:25.171886 30030 leader_election.cc:288] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:25.171984 30042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:25.173717 30042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [term 1 LEADER]: Becoming Leader. State: Replica: 9422c1f3e2b94f4c8b3227453de89238, State: Running, Role: LEADER
I20260809 15:06:25.174067 30030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:25.174367 30042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9422c1f3e2b94f4c8b3227453de89238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41411 } }
I20260809 15:06:25.177030 30053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9422c1f3e2b94f4c8b3227453de89238. Latest consensus state: current_term: 1 leader_uuid: "9422c1f3e2b94f4c8b3227453de89238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9422c1f3e2b94f4c8b3227453de89238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41411 } } }
I20260809 15:06:25.177944 30053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:25.177632 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9422c1f3e2b94f4c8b3227453de89238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9422c1f3e2b94f4c8b3227453de89238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41411 } } }
I20260809 15:06:25.178494 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:25.185446 30061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:25.189571 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:25.172584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:50101 (local address 127.0.0.1:41411)
0809 15:06:25.172731 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:25.172738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:25.173157 (+ 419us) server_negotiation.cc:407] Connection header received
0809 15:06:25.173374 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:25.173378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:25.173464 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:25.173569 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:25.175255 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:25.176151 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:25.177920 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:25.178668 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:25.178796 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:25.182296 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:25.182303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:25.182309 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:25.182352 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:25.185927 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:25.186863 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:25.186869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:25.186873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:25.187021 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:25.187699 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:25.187704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:25.187707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:25.188121 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:25.188358 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:25.189030 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 15:06:25.189220 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 15:06:25.190570 30061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:25.198731 30061 catalog_manager.cc:1347] Generated new cluster ID: 9c03fbe0db3e4046ac44c0c42274540f
I20260809 15:06:25.198828 30061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:25.219132 30061 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:25.220252 30061 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:25.227756 30061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9422c1f3e2b94f4c8b3227453de89238: Generated new TSK 0
I20260809 15:06:25.228396 30061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:25.235705 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 29917
I20260809 15:06:25.236869 30021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:25.324536 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 29917
W20260809 15:06:25.346127 29895 connection.cc:535] client connection to 127.0.0.1:41411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41411): Connection reset by peer (error 104)
2026-08-09T15:06:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:25 dist-test-slave-q9h9 krb5kdc[30085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:25 dist-test-slave-q9h9 krb5kdc[30085](info): set up 2 sockets
Aug 09 15:06:25 dist-test-slave-q9h9 krb5kdc[30085](info): commencing operation
krb5kdc: starting...
W20260809 15:06:26.545739 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:26 dist-test-slave-q9h9 krb5kdc[30085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:26.790151 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:27.102453 30105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:27.102721 30105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:27.102864 30105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:27.109685 30105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:27.109799 30105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:27.109876 30105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:27.109961 30105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:27.110037 30105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:27.119238 30105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:27.120347 30105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:27.121522 30105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:27.132309 30111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:27.132400 30114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:27.132421 30112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:27.133107 30105 server_base.cc:1034] running on GCE node
I20260809 15:06:27.133805 30105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:27.135089 30105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:27.136317 30105 hybrid_clock.cc:648] HybridClock initialized: now 1786287987136265 us; error 46 us; skew 500 ppm
I20260809 15:06:27.136649 30105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:27 dist-test-slave-q9h9 krb5kdc[30085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:27.139654 30105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:27.140939 30105 webserver.cc:460] Webserver started at http://127.0.0.1:46763/ using document root <none> and password file <none>
I20260809 15:06:27.141547 30105 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:27.141664 30105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:27.141954 30105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:27.144744 30105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "1d27aef7660c4bb8ad14008cb3aa130f"
format_stamp: "Formatted at 2026-08-09 15:06:27 on dist-test-slave-q9h9"
server_key: "69fe2812d22bce98e5036d2627964218"
server_key_iv: "2e918f3a7e3310acc924cb52ddf18a79"
server_key_version: "encryptionkey@0"
I20260809 15:06:27.145469 30105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "1d27aef7660c4bb8ad14008cb3aa130f"
format_stamp: "Formatted at 2026-08-09 15:06:27 on dist-test-slave-q9h9"
server_key: "69fe2812d22bce98e5036d2627964218"
server_key_iv: "2e918f3a7e3310acc924cb52ddf18a79"
server_key_version: "encryptionkey@0"
I20260809 15:06:27.150204 30105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 15:06:27.153326 30128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:27.154515 30105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:27.154672 30105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "1d27aef7660c4bb8ad14008cb3aa130f"
format_stamp: "Formatted at 2026-08-09 15:06:27 on dist-test-slave-q9h9"
server_key: "69fe2812d22bce98e5036d2627964218"
server_key_iv: "2e918f3a7e3310acc924cb52ddf18a79"
server_key_version: "encryptionkey@0"
I20260809 15:06:27.154850 30105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:27.168571 30105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:27.172389 30105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:27.172621 30105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:27.189185 30105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46815
I20260809 15:06:27.189215 30215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46815 every 8 connection(s)
I20260809 15:06:27.190419 30105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:27.191123 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30105
I20260809 15:06:27.191318 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:27.191850 28394 external_mini_cluster.cc:1428] Setting key 43d40238f801e4b2cf29470c0dbc6832
I20260809 15:06:27.194497 30218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:27.203763 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:27.193517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:44051 (local address 127.0.0.1:46815)
0809 15:06:27.194230 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:27.194241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:27.194288 (+ 47us) server_negotiation.cc:407] Connection header received
0809 15:06:27.195885 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:27.195905 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:27.196234 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:27.196569 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:27.197150 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:27.198479 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:27.199784 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:27.200409 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:27.200569 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:27.201697 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:27.201705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:27.201717 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:27.202267 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:27.202355 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:27.202381 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:27.202539 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:27.202810 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 15:06:27.202940 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":295,"threads_started":1}
I20260809 15:06:27.204439 30218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f: Bootstrap starting.
I20260809 15:06:27.207033 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:27.207434 30218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:27.208354 30218 log.cc:824] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:27.210975 30218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f: No bootstrap required, opened a new log
I20260809 15:06:27.217432 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:27.210943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:45127 (local address 127.0.0.1:46815)
0809 15:06:27.211097 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:27.211102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:27.211638 (+ 536us) server_negotiation.cc:407] Connection header received
0809 15:06:27.211812 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:27.211816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:27.211866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:27.211936 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:27.212532 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:27.213258 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:27.215123 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:27.215593 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:27.215684 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:27.216734 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:27.216739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:27.216745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:27.216760 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:27.216810 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:27.216823 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:27.216905 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:27.217101 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 15:06:27.217137 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:06:27.219641 30218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260809 15:06:27.219954 30218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:27.220052 30218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d27aef7660c4bb8ad14008cb3aa130f, State: Initialized, Role: FOLLOWER
I20260809 15:06:27.220551 30218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260809 15:06:27.220770 30218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:27.220893 30218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:27.221041 30218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:27.223057 30218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260809 15:06:27.223492 30218 leader_election.cc:302] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d27aef7660c4bb8ad14008cb3aa130f; no voters:
I20260809 15:06:27.224018 30218 leader_election.cc:288] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:27.224117 30234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:27.224707 30234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [term 1 LEADER]: Becoming Leader. State: Replica: 1d27aef7660c4bb8ad14008cb3aa130f, State: Running, Role: LEADER
I20260809 15:06:27.225193 30234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260809 15:06:27.225796 30218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:27.227747 30235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260809 15:06:27.227869 30238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d27aef7660c4bb8ad14008cb3aa130f. Latest consensus state: current_term: 1 leader_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27aef7660c4bb8ad14008cb3aa130f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260809 15:06:27.228129 30235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:27.228184 30238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:27.230877 30245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:27.234789 30245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:27.242615 30245 catalog_manager.cc:1347] Generated new cluster ID: e2646306cfca40b8910329669671b082
I20260809 15:06:27.242714 30245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:27.272751 30245 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:27.274504 30245 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:27.280830 30245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d27aef7660c4bb8ad14008cb3aa130f: Generated new TSK 0
I20260809 15:06:27.281773 30245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:27.287912 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30105
W20260809 15:06:27.289088 30209 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:44051
W20260809 15:06:27.289844 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:27.290007 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30105
W20260809 15:06:27.311445 30082 connection.cc:535] client connection to 127.0.0.1:46815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46815): Connection reset by peer (error 104)
2026-08-09T15:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:27 dist-test-slave-q9h9 krb5kdc[30267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:27 dist-test-slave-q9h9 krb5kdc[30267](info): set up 2 sockets
Aug 09 15:06:27 dist-test-slave-q9h9 krb5kdc[30267](info): commencing operation
krb5kdc: starting...
W20260809 15:06:28.506814 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.003s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:28 dist-test-slave-q9h9 krb5kdc[30267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:28.740536 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:29.047435 30289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:29.047698 30289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:29.047840 30289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:29.054720 30289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:29.054843 30289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:29.054919 30289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:29.055001 30289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:29.055058 30289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:29.064270 30289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:29.065389 30289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:29.066619 30289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:29.077581 30301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:29.077786 30303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:29.077880 30305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:29.078811 30289 server_base.cc:1034] running on GCE node
I20260809 15:06:29.079474 30289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:29.080662 30289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:29.081869 30289 hybrid_clock.cc:648] HybridClock initialized: now 1786287989081813 us; error 45 us; skew 500 ppm
I20260809 15:06:29.082221 30289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:29 dist-test-slave-q9h9 krb5kdc[30267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:29.088855 30289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:29.090150 30289 webserver.cc:460] Webserver started at http://127.0.0.1:34899/ using document root <none> and password file <none>
I20260809 15:06:29.090734 30289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:29.090831 30289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:29.091075 30289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:29.093772 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "1e55247d12da42ffaa466091fb03dded"
format_stamp: "Formatted at 2026-08-09 15:06:29 on dist-test-slave-q9h9"
server_key: "12bf7b6ce2d5bc7c838cac85956dcd98"
server_key_iv: "50b873d859825a9de725cc58e7415c07"
server_key_version: "encryptionkey@0"
I20260809 15:06:29.094560 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "1e55247d12da42ffaa466091fb03dded"
format_stamp: "Formatted at 2026-08-09 15:06:29 on dist-test-slave-q9h9"
server_key: "12bf7b6ce2d5bc7c838cac85956dcd98"
server_key_iv: "50b873d859825a9de725cc58e7415c07"
server_key_version: "encryptionkey@0"
I20260809 15:06:29.099255 30289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 15:06:29.102372 30322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:29.103551 30289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 15:06:29.103699 30289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "1e55247d12da42ffaa466091fb03dded"
format_stamp: "Formatted at 2026-08-09 15:06:29 on dist-test-slave-q9h9"
server_key: "12bf7b6ce2d5bc7c838cac85956dcd98"
server_key_iv: "50b873d859825a9de725cc58e7415c07"
server_key_version: "encryptionkey@0"
I20260809 15:06:29.103858 30289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:29.109162 30289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:29.112679 30289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:29.112944 30289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:29.130656 30289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36249
I20260809 15:06:29.130716 30401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36249 every 8 connection(s)
I20260809 15:06:29.131912 30289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:29.135354 30403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:29.139638 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30289
I20260809 15:06:29.139797 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:29.140197 28394 external_mini_cluster.cc:1428] Setting key 38955146c8ff9656a9a686afbf47e7b2
I20260809 15:06:29.146939 30403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded: Bootstrap starting.
I20260809 15:06:29.148783 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:29.141806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:32819 (local address 127.0.0.1:36249)
0809 15:06:29.142646 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:29.142660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:29.142709 (+ 49us) server_negotiation.cc:407] Connection header received
0809 15:06:29.145126 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:29.145152 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:29.145568 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:29.145971 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:29.146936 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:29.146944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:29.146959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:29.146977 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:29.147046 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:29.147079 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:29.147300 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:29.147447 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 15:06:29.147622 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":338,"threads_started":1}
I20260809 15:06:29.150369 30403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:29.151477 30403 log.cc:824] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:29.152130 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:29.154299 30403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded: No bootstrap required, opened a new log
I20260809 15:06:29.159684 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:29.157170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:33755 (local address 127.0.0.1:36249)
0809 15:06:29.157336 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:29.157341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:29.157942 (+ 601us) server_negotiation.cc:407] Connection header received
0809 15:06:29.158088 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:29.158093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:29.158155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:29.158254 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:29.159051 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:29.159057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:29.159062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:29.159082 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:29.159127 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:29.159149 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:29.159253 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:29.159335 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 15:06:29.159395 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 15:06:29.162842 30403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e55247d12da42ffaa466091fb03dded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260809 15:06:29.163147 30403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:29.163216 30403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e55247d12da42ffaa466091fb03dded, State: Initialized, Role: FOLLOWER
I20260809 15:06:29.163666 30403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e55247d12da42ffaa466091fb03dded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260809 15:06:29.163888 30403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:29.163983 30403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:29.164100 30403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:29.166112 30403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e55247d12da42ffaa466091fb03dded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260809 15:06:29.166517 30403 leader_election.cc:302] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e55247d12da42ffaa466091fb03dded; no voters:
I20260809 15:06:29.166966 30403 leader_election.cc:288] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:29.167084 30418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:29.167658 30418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [term 1 LEADER]: Becoming Leader. State: Replica: 1e55247d12da42ffaa466091fb03dded, State: Running, Role: LEADER
I20260809 15:06:29.168181 30418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e55247d12da42ffaa466091fb03dded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260809 15:06:29.168876 30403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:29.170779 30419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e55247d12da42ffaa466091fb03dded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e55247d12da42ffaa466091fb03dded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260809 15:06:29.171209 30419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:29.171908 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e55247d12da42ffaa466091fb03dded. Latest consensus state: current_term: 1 leader_uuid: "1e55247d12da42ffaa466091fb03dded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e55247d12da42ffaa466091fb03dded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260809 15:06:29.172196 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:29.172657 30430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:29.177579 30430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:29.186035 30430 catalog_manager.cc:1347] Generated new cluster ID: 4392d3a6118c40ca9cbf90b824bb6198
I20260809 15:06:29.186138 30430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:29.213820 30430 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:29.214287 30430 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:29.235553 30430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e55247d12da42ffaa466091fb03dded: Generated new TSK 0
I20260809 15:06:29.236431 30430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:29.247246 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30289
W20260809 15:06:29.248369 30394 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:32819
W20260809 15:06:29.249047 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30289: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:29.249202 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30289
2026-08-09T15:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:29 dist-test-slave-q9h9 krb5kdc[30451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:29 dist-test-slave-q9h9 krb5kdc[30451](info): set up 2 sockets
Aug 09 15:06:29 dist-test-slave-q9h9 krb5kdc[30451](info): commencing operation
krb5kdc: starting...
W20260809 15:06:30.469897 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:30 dist-test-slave-q9h9 krb5kdc[30451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:30.702256 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:31.006107 30481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:31.006341 30481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:31.006470 30481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:31.013217 30481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:31.013304 30481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:31.013381 30481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:31.013458 30481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:31.013516 30481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:31.022544 30481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:31.023588 30481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:31.024698 30481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:31.035243 30494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:31.035238 30490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:31.035279 30489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:31.036052 30481 server_base.cc:1034] running on GCE node
I20260809 15:06:31.036685 30481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:31.037871 30481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:31.039059 30481 hybrid_clock.cc:648] HybridClock initialized: now 1786287991039010 us; error 45 us; skew 500 ppm
I20260809 15:06:31.039356 30481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:31 dist-test-slave-q9h9 krb5kdc[30451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:31.042204 30481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:31.043463 30481 webserver.cc:460] Webserver started at http://127.0.0.1:35969/ using document root <none> and password file <none>
I20260809 15:06:31.044042 30481 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:31.044157 30481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:31.044423 30481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:31.047117 30481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3"
format_stamp: "Formatted at 2026-08-09 15:06:31 on dist-test-slave-q9h9"
server_key: "3d6a0f47a8856eefd5d325cb4b2a1464"
server_key_iv: "394b51a3a34f0ea4d47dff9d1f5fa8f5"
server_key_version: "encryptionkey@0"
I20260809 15:06:31.047817 30481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3"
format_stamp: "Formatted at 2026-08-09 15:06:31 on dist-test-slave-q9h9"
server_key: "3d6a0f47a8856eefd5d325cb4b2a1464"
server_key_iv: "394b51a3a34f0ea4d47dff9d1f5fa8f5"
server_key_version: "encryptionkey@0"
I20260809 15:06:31.052407 30481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:06:31.055522 30509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:31.056645 30481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:06:31.056808 30481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3"
format_stamp: "Formatted at 2026-08-09 15:06:31 on dist-test-slave-q9h9"
server_key: "3d6a0f47a8856eefd5d325cb4b2a1464"
server_key_iv: "394b51a3a34f0ea4d47dff9d1f5fa8f5"
server_key_version: "encryptionkey@0"
I20260809 15:06:31.056978 30481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:31.074165 30481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:31.077538 30481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:31.077767 30481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:31.095234 30481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38923
I20260809 15:06:31.095273 30596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38923 every 8 connection(s)
I20260809 15:06:31.096380 30481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:31.099887 30602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:31.101480 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30481
I20260809 15:06:31.101650 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:31.102101 28394 external_mini_cluster.cc:1428] Setting key 1740256d82af44c5fff90fe161003e4e
I20260809 15:06:31.113163 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3: Bootstrap starting.
Aug 09 15:06:31 dist-test-slave-q9h9 krb5kdc[30451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 15:06:31.116876 30602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:31.117939 30602 log.cc:824] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:31.120538 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3: No bootstrap required, opened a new log
I20260809 15:06:31.124065 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:31.103801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:50349 (local address 127.0.0.1:38923)
0809 15:06:31.104549 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:31.104559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:31.104600 (+ 41us) server_negotiation.cc:407] Connection header received
0809 15:06:31.106143 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:31.106163 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:31.106520 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:31.106817 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:31.108491 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:31.109720 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:31.111504 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:31.112121 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:31.112235 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:31.117221 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:31.117228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:31.117243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:31.117282 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:31.120044 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:31.120936 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:31.120942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:31.120947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:31.121119 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:31.121601 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:31.121604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:31.121606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:31.122128 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:31.122382 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:31.122873 (+ 491us) server_negotiation.cc:299] Negotiation successful
0809 15:06:31.123127 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":336,"threads_started":1}
I20260809 15:06:31.127668 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:31.129031 30602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260809 15:06:31.129352 30602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:31.129438 30602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc62bcec08a4cca9a91a2b8e75cc8c3, State: Initialized, Role: FOLLOWER
I20260809 15:06:31.129945 30602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260809 15:06:31.130239 30602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:31.130376 30602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:31.130541 30602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:31.132962 30602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260809 15:06:31.133399 30602 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fc62bcec08a4cca9a91a2b8e75cc8c3; no voters:
I20260809 15:06:31.133932 30602 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:31.134097 30619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:31.135160 30619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc62bcec08a4cca9a91a2b8e75cc8c3, State: Running, Role: LEADER
I20260809 15:06:31.135712 30619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260809 15:06:31.136067 30602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:31.138391 30620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } } }
I20260809 15:06:31.138890 30620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:31.138468 30623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc62bcec08a4cca9a91a2b8e75cc8c3. Latest consensus state: current_term: 1 leader_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc62bcec08a4cca9a91a2b8e75cc8c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } } }
I20260809 15:06:31.139467 30623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:31.146835 30627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:31.148864 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:31.131814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:42609 (local address 127.0.0.1:38923)
0809 15:06:31.131978 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:31.131983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:31.132381 (+ 398us) server_negotiation.cc:407] Connection header received
0809 15:06:31.132487 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:31.132492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:31.132557 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:31.132648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:31.134174 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:31.135356 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:31.137088 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:31.137815 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:31.137939 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:31.140943 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:31.140949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:31.140954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:31.140988 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:31.144645 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:31.145631 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:31.145637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:31.145641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:31.145787 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:31.146449 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:31.146455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:31.146458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:31.146845 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:31.148123 (+ 1278us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:31.148312 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 15:06:31.148489 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 15:06:31.151917 30627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:31.160187 30627 catalog_manager.cc:1347] Generated new cluster ID: 63c870e0e08943a69a048ca18de8ed5c
I20260809 15:06:31.160296 30627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:31.175561 30627 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:31.176843 30627 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:31.194283 30627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fc62bcec08a4cca9a91a2b8e75cc8c3: Generated new TSK 0
I20260809 15:06:31.195147 30627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:31.203027 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30481
I20260809 15:06:31.204457 30590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:31.293299 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30481
W20260809 15:06:31.315341 30448 connection.cc:535] client connection to 127.0.0.1:38923 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38923): Connection reset by peer (error 104)
2026-08-09T15:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:31 dist-test-slave-q9h9 krb5kdc[30655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:31 dist-test-slave-q9h9 krb5kdc[30655](info): set up 2 sockets
Aug 09 15:06:31 dist-test-slave-q9h9 krb5kdc[30655](info): commencing operation
krb5kdc: starting...
W20260809 15:06:32.521598 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:32 dist-test-slave-q9h9 krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:32.766810 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:33.072106 30670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:33.072346 30670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:33.072440 30670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:33.079150 30670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:33.079242 30670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:33.079300 30670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:33.079360 30670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:33.079401 30670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:33.088346 30670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:33.089394 30670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:33.090534 30670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:33.100881 30677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:33.100970 30679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:33.100934 30682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:33.101781 30670 server_base.cc:1034] running on GCE node
I20260809 15:06:33.102519 30670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:33.103741 30670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:33.104939 30670 hybrid_clock.cc:648] HybridClock initialized: now 1786287993104891 us; error 49 us; skew 500 ppm
I20260809 15:06:33.105242 30670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:33 dist-test-slave-q9h9 krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:33.108119 30670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:33.109423 30670 webserver.cc:460] Webserver started at http://127.0.0.1:43149/ using document root <none> and password file <none>
I20260809 15:06:33.110004 30670 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:33.110172 30670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:33.110455 30670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:33.113087 30670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "3313dc8f0c524548a559e42ed8be3543"
format_stamp: "Formatted at 2026-08-09 15:06:33 on dist-test-slave-q9h9"
server_key: "51893cb8a78abee6b5048bbfcf0d218d"
server_key_iv: "f53efdbdf44bf6677adc1e0a247c0a68"
server_key_version: "encryptionkey@0"
I20260809 15:06:33.113777 30670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "3313dc8f0c524548a559e42ed8be3543"
format_stamp: "Formatted at 2026-08-09 15:06:33 on dist-test-slave-q9h9"
server_key: "51893cb8a78abee6b5048bbfcf0d218d"
server_key_iv: "f53efdbdf44bf6677adc1e0a247c0a68"
server_key_version: "encryptionkey@0"
I20260809 15:06:33.118516 30670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 15:06:33.121559 30695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:33.122716 30670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:33.122861 30670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "3313dc8f0c524548a559e42ed8be3543"
format_stamp: "Formatted at 2026-08-09 15:06:33 on dist-test-slave-q9h9"
server_key: "51893cb8a78abee6b5048bbfcf0d218d"
server_key_iv: "f53efdbdf44bf6677adc1e0a247c0a68"
server_key_version: "encryptionkey@0"
I20260809 15:06:33.123020 30670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:33.149241 30670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:33.152565 30670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:33.152804 30670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:33.169925 30670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36997
I20260809 15:06:33.169975 30780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36997 every 8 connection(s)
I20260809 15:06:33.171124 30670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:33.174425 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:33.175264 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30670
I20260809 15:06:33.175429 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:33.175833 28394 external_mini_cluster.cc:1428] Setting key 7ba316928da094cc9f2ea195e5270ba7
I20260809 15:06:33.187016 30782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543: Bootstrap starting.
I20260809 15:06:33.188561 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:33.177415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:46893 (local address 127.0.0.1:36997)
0809 15:06:33.178149 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:33.178160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:33.178203 (+ 43us) server_negotiation.cc:407] Connection header received
0809 15:06:33.179907 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:33.179930 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:33.180304 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:33.180599 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:33.181224 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:33.182478 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:33.184349 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:33.185002 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:33.185122 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:33.186407 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:33.186417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:33.186434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:33.187046 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:33.187114 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:33.187140 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:33.187286 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:33.187530 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 15:06:33.187680 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":317,"threads_started":1}
I20260809 15:06:33.190264 30782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:33.191404 30782 log.cc:824] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:33.192205 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:33.194257 30782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543: No bootstrap required, opened a new log
I20260809 15:06:33.203421 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:33.197102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:43629 (local address 127.0.0.1:36997)
0809 15:06:33.197271 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:33.197276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:33.197733 (+ 457us) server_negotiation.cc:407] Connection header received
0809 15:06:33.197895 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:33.197898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:33.197955 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:33.198049 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:33.198608 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:33.199331 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:33.201116 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:33.201599 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:33.201725 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:33.202770 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:33.202782 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:33.202785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:33.202800 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:33.202850 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:33.202864 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:33.202946 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:33.203121 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 15:06:33.203156 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:06:33.203800 30782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3313dc8f0c524548a559e42ed8be3543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36997 } }
I20260809 15:06:33.204147 30782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:33.204241 30782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3313dc8f0c524548a559e42ed8be3543, State: Initialized, Role: FOLLOWER
I20260809 15:06:33.204780 30782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3313dc8f0c524548a559e42ed8be3543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36997 } }
I20260809 15:06:33.204994 30782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:33.205129 30782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:33.205273 30782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:33.207306 30782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3313dc8f0c524548a559e42ed8be3543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36997 } }
I20260809 15:06:33.207732 30782 leader_election.cc:302] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3313dc8f0c524548a559e42ed8be3543; no voters:
I20260809 15:06:33.208278 30782 leader_election.cc:288] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:33.208420 30800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:33.209089 30800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [term 1 LEADER]: Becoming Leader. State: Replica: 3313dc8f0c524548a559e42ed8be3543, State: Running, Role: LEADER
I20260809 15:06:33.209568 30800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3313dc8f0c524548a559e42ed8be3543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36997 } }
I20260809 15:06:33.210117 30782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:33.212055 30804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3313dc8f0c524548a559e42ed8be3543. Latest consensus state: current_term: 1 leader_uuid: "3313dc8f0c524548a559e42ed8be3543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3313dc8f0c524548a559e42ed8be3543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36997 } } }
I20260809 15:06:33.212574 30804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:33.212517 30802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3313dc8f0c524548a559e42ed8be3543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3313dc8f0c524548a559e42ed8be3543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36997 } } }
I20260809 15:06:33.212832 30802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:33.213465 30812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:33.218783 30812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:33.228549 30812 catalog_manager.cc:1347] Generated new cluster ID: a2936a8b79f64ae89b0372196f480960
I20260809 15:06:33.228641 30812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:33.252467 30812 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:33.254061 30812 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:33.267189 30812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3313dc8f0c524548a559e42ed8be3543: Generated new TSK 0
I20260809 15:06:33.268162 30812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:33.286316 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30670
W20260809 15:06:33.287298 30775 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:46893
W20260809 15:06:33.287952 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:33.288079 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30670
W20260809 15:06:33.316951 30653 connection.cc:535] client connection to 127.0.0.1:36997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36997): Connection reset by peer (error 104)
2026-08-09T15:06:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:33 dist-test-slave-q9h9 krb5kdc[30839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:33 dist-test-slave-q9h9 krb5kdc[30839](info): set up 2 sockets
Aug 09 15:06:33 dist-test-slave-q9h9 krb5kdc[30839](info): commencing operation
krb5kdc: starting...
W20260809 15:06:34.536145 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:34 dist-test-slave-q9h9 krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 15:06:34.771018 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:35.077324 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:35.077581 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:35.077720 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:35.084575 30859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:35.084679 30859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:35.084753 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:35.084830 30859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:35.084890 30859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:35.093942 30859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:35.095005 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:35.096114 30859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:35.106899 30871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:35.106931 30867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:35.107453 30866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:35.107890 30859 server_base.cc:1034] running on GCE node
I20260809 15:06:35.108574 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:35.109789 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:35.110991 30859 hybrid_clock.cc:648] HybridClock initialized: now 1786287995110937 us; error 48 us; skew 500 ppm
I20260809 15:06:35.111304 30859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:35 dist-test-slave-q9h9 krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:35.114120 30859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 15:06:35.115350 30859 webserver.cc:460] Webserver started at http://127.0.0.1:46537/ using document root <none> and password file <none>
I20260809 15:06:35.115909 30859 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:35.116004 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:35.116232 30859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:35.118942 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "4522f36c97b94744a68297455a61edbe"
format_stamp: "Formatted at 2026-08-09 15:06:35 on dist-test-slave-q9h9"
server_key: "421f3676f8f4ee4cd45d418c7bcaece2"
server_key_iv: "031f96774877931d32052ae3007ae4e7"
server_key_version: "encryptionkey@0"
I20260809 15:06:35.119593 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "4522f36c97b94744a68297455a61edbe"
format_stamp: "Formatted at 2026-08-09 15:06:35 on dist-test-slave-q9h9"
server_key: "421f3676f8f4ee4cd45d418c7bcaece2"
server_key_iv: "031f96774877931d32052ae3007ae4e7"
server_key_version: "encryptionkey@0"
I20260809 15:06:35.124207 30859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 15:06:35.127316 30882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:35.128492 30859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:35.128621 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "4522f36c97b94744a68297455a61edbe"
format_stamp: "Formatted at 2026-08-09 15:06:35 on dist-test-slave-q9h9"
server_key: "421f3676f8f4ee4cd45d418c7bcaece2"
server_key_iv: "031f96774877931d32052ae3007ae4e7"
server_key_version: "encryptionkey@0"
I20260809 15:06:35.128788 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:35.133750 30859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:35.137360 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:35.137594 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:35.153036 30859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41125
I20260809 15:06:35.153090 30981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41125 every 8 connection(s)
I20260809 15:06:35.154436 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:35.157747 30984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:35.160706 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 30859
I20260809 15:06:35.160846 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:35.161273 28394 external_mini_cluster.cc:1428] Setting key 68351c5cd2dec466fe776ba651e0c6c8
I20260809 15:06:35.168206 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:35.162904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:40395 (local address 127.0.0.1:41125)
0809 15:06:35.163649 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:35.163659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:35.163697 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:06:35.165272 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:35.165293 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:35.165596 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:35.165903 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:35.166867 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:35.166873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:35.166884 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:35.166905 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:35.166962 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:35.166980 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:35.167145 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:35.167240 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 15:06:35.167386 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":311,"threads_started":1}
I20260809 15:06:35.169091 30984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe: Bootstrap starting.
I20260809 15:06:35.171663 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:35.172145 30984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:35.173087 30984 log.cc:824] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:35.175597 30984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe: No bootstrap required, opened a new log
I20260809 15:06:35.177901 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:35.175810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:43919 (local address 127.0.0.1:41125)
0809 15:06:35.175958 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:35.175963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:35.176290 (+ 327us) server_negotiation.cc:407] Connection header received
0809 15:06:35.176400 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:35.176405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:35.176456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:35.176551 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:35.177309 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:35.177315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:35.177320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:35.177337 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:35.177384 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:35.177402 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:35.177514 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:35.177570 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 15:06:35.177624 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:35.183125 30984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522f36c97b94744a68297455a61edbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41125 } }
I20260809 15:06:35.183425 30984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:35.183514 30984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4522f36c97b94744a68297455a61edbe, State: Initialized, Role: FOLLOWER
I20260809 15:06:35.183997 30984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522f36c97b94744a68297455a61edbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41125 } }
I20260809 15:06:35.184206 30984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:35.184300 30984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:35.184422 30984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:35.186667 30984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522f36c97b94744a68297455a61edbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41125 } }
I20260809 15:06:35.187068 30984 leader_election.cc:302] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4522f36c97b94744a68297455a61edbe; no voters:
I20260809 15:06:35.187567 30984 leader_election.cc:288] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:35.187665 31001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:35.188271 31001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [term 1 LEADER]: Becoming Leader. State: Replica: 4522f36c97b94744a68297455a61edbe, State: Running, Role: LEADER
I20260809 15:06:35.188745 31001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522f36c97b94744a68297455a61edbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41125 } }
I20260809 15:06:35.189316 30984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:35.191412 31002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4522f36c97b94744a68297455a61edbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522f36c97b94744a68297455a61edbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41125 } } }
I20260809 15:06:35.191557 31003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4522f36c97b94744a68297455a61edbe. Latest consensus state: current_term: 1 leader_uuid: "4522f36c97b94744a68297455a61edbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522f36c97b94744a68297455a61edbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41125 } } }
I20260809 15:06:35.191855 31002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:35.191911 31003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:35.192868 31013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:35.204492 31013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:35.212738 31013 catalog_manager.cc:1347] Generated new cluster ID: 96e66c1c4ccf49a19fddf8e59de79a0e
I20260809 15:06:35.212844 31013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:35.223341 31013 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:35.223675 31013 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:35.229701 31013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4522f36c97b94744a68297455a61edbe: Generated new TSK 0
I20260809 15:06:35.230419 31013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:35.244112 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30859
W20260809 15:06:35.245116 30975 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:40395
W20260809 15:06:35.245796 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 30859: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:35.245922 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 30859
2026-08-09T15:06:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:35 dist-test-slave-q9h9 krb5kdc[31036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:35 dist-test-slave-q9h9 krb5kdc[31036](info): set up 2 sockets
Aug 09 15:06:35 dist-test-slave-q9h9 krb5kdc[31036](info): commencing operation
krb5kdc: starting...
W20260809 15:06:36.466090 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:36 dist-test-slave-q9h9 krb5kdc[31036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:36.698122 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38397
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:37.002437 31053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:37.002677 31053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:37.002802 31053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:37.009678 31053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:37.009769 31053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:37.009841 31053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:37.009914 31053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:37.009969 31053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:37.018989 31053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:38397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:37.019982 31053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:37.021145 31053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:37.031708 31065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:37.031751 31068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:37.031777 31063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:37.032853 31053 server_base.cc:1034] running on GCE node
I20260809 15:06:37.033573 31053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:37.034782 31053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:37.035976 31053 hybrid_clock.cc:648] HybridClock initialized: now 1786287997035938 us; error 32 us; skew 500 ppm
I20260809 15:06:37.036280 31053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:37 dist-test-slave-q9h9 krb5kdc[31036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:37.039106 31053 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:37.040401 31053 webserver.cc:460] Webserver started at http://127.27.186.190:36007/ using document root <none> and password file <none>
I20260809 15:06:37.040973 31053 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:37.041079 31053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:37.041348 31053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:37.043972 31053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "211c6772c96d48c1b81d035f3ad6e63e"
format_stamp: "Formatted at 2026-08-09 15:06:37 on dist-test-slave-q9h9"
server_key: "aa9709d30d0e4df9591bfca7c962502e"
server_key_iv: "08f493f7bf9efde28d05761e586a346b"
server_key_version: "encryptionkey@0"
I20260809 15:06:37.044651 31053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "211c6772c96d48c1b81d035f3ad6e63e"
format_stamp: "Formatted at 2026-08-09 15:06:37 on dist-test-slave-q9h9"
server_key: "aa9709d30d0e4df9591bfca7c962502e"
server_key_iv: "08f493f7bf9efde28d05761e586a346b"
server_key_version: "encryptionkey@0"
I20260809 15:06:37.049194 31053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260809 15:06:37.052318 31084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:37.053436 31053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:06:37.053598 31053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "211c6772c96d48c1b81d035f3ad6e63e"
format_stamp: "Formatted at 2026-08-09 15:06:37 on dist-test-slave-q9h9"
server_key: "aa9709d30d0e4df9591bfca7c962502e"
server_key_iv: "08f493f7bf9efde28d05761e586a346b"
server_key_version: "encryptionkey@0"
I20260809 15:06:37.053774 31053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:37.067967 31053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:37.071834 31053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:37.072068 31053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:37.087556 31053 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:38397
I20260809 15:06:37.087599 31164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:38397 every 8 connection(s)
I20260809 15:06:37.088708 31053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:37.091913 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:37.096328 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31053
I20260809 15:06:37.096481 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:37.096872 28394 external_mini_cluster.cc:1428] Setting key 80bd23f9272467d37331d68de3487a04
I20260809 15:06:37.102932 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e: Bootstrap starting.
I20260809 15:06:37.106318 31167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:37.107338 31167 log.cc:824] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:37.109933 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e: No bootstrap required, opened a new log
Aug 09 15:06:37 dist-test-slave-q9h9 krb5kdc[31036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:37.117278 31167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211c6772c96d48c1b81d035f3ad6e63e" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38397 } }
I20260809 15:06:37.117591 31167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:37.117697 31167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211c6772c96d48c1b81d035f3ad6e63e, State: Initialized, Role: FOLLOWER
I20260809 15:06:37.118321 31167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211c6772c96d48c1b81d035f3ad6e63e" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38397 } }
I20260809 15:06:37.118546 31167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:37.118685 31167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:37.118849 31167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:37.119001 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:37.098371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48737 (local address 127.27.186.190:38397)
0809 15:06:37.099222 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:37.099238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:37.099286 (+ 48us) server_negotiation.cc:407] Connection header received
0809 15:06:37.101597 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:37.101623 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:37.102095 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:37.102497 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:37.103789 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:37.105709 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:37.107001 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:37.108032 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:37.108199 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:37.112067 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:37.112074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:37.112089 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:37.112123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:37.115056 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:37.115780 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:37.115787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:37.115792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:37.115955 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:37.116416 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:37.116419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:37.116421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:37.116891 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:37.117176 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:37.117662 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 15:06:37.117944 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":387,"threads_started":1}
I20260809 15:06:37.121305 31167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211c6772c96d48c1b81d035f3ad6e63e" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38397 } }
I20260809 15:06:37.121860 31167 leader_election.cc:302] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211c6772c96d48c1b81d035f3ad6e63e; no voters:
I20260809 15:06:37.122531 31167 leader_election.cc:288] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:37.122674 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:37.122637 31177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:37.124563 31167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:37.130910 31177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [term 1 LEADER]: Becoming Leader. State: Replica: 211c6772c96d48c1b81d035f3ad6e63e, State: Running, Role: LEADER
I20260809 15:06:37.131413 31177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211c6772c96d48c1b81d035f3ad6e63e" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38397 } }
I20260809 15:06:37.134342 31180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211c6772c96d48c1b81d035f3ad6e63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211c6772c96d48c1b81d035f3ad6e63e" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38397 } } }
I20260809 15:06:37.134755 31180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:37.141412 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211c6772c96d48c1b81d035f3ad6e63e. Latest consensus state: current_term: 1 leader_uuid: "211c6772c96d48c1b81d035f3ad6e63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211c6772c96d48c1b81d035f3ad6e63e" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38397 } } }
I20260809 15:06:37.141835 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:37.147630 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:37.131442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44501 (local address 127.27.186.190:38397)
0809 15:06:37.131597 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:37.131603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:37.131975 (+ 372us) server_negotiation.cc:407] Connection header received
0809 15:06:37.132150 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:37.132154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:37.132212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:37.132315 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:37.134103 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:37.134932 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:37.136904 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:37.137626 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:37.137738 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:37.141090 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:37.141097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:37.141102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:37.141140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:37.144736 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:37.145582 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:37.145588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:37.145592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:37.145730 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:37.146139 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:37.146144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:37.146147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:37.146515 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:37.146794 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:37.147121 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 15:06:37.147298 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260809 15:06:37.149797 31214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:06:37.149894 31214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:06:37.150004 31211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:37.151600 31211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:37.158917 31211 catalog_manager.cc:1347] Generated new cluster ID: 6e77772dae53472598e8da694338332e
I20260809 15:06:37.159011 31211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:37.170557 31211 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:37.172226 31211 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:37.180811 31211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211c6772c96d48c1b81d035f3ad6e63e: Generated new TSK 0
I20260809 15:06:37.181643 31211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:37.186441 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31053
I20260809 15:06:37.188645 31160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:37.273476 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31053
W20260809 15:06:37.294965 31033 connection.cc:535] client connection to 127.27.186.190:38397 recv error: Network error: failed to read from TLS socket (remote: 127.27.186.190:38397): Connection reset by peer (error 104)
2026-08-09T15:06:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:37 dist-test-slave-q9h9 krb5kdc[31228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:37 dist-test-slave-q9h9 krb5kdc[31228](info): set up 2 sockets
Aug 09 15:06:37 dist-test-slave-q9h9 krb5kdc[31228](info): commencing operation
krb5kdc: starting...
W20260809 15:06:38.479291 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.005s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:38 dist-test-slave-q9h9 krb5kdc[31228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:38.714804 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42141
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:39.021060 31255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:39.021327 31255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:39.021457 31255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:39.028309 31255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:39.028417 31255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:39.028491 31255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:39.028574 31255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:39.028628 31255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:39.037691 31255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:42141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:39.038748 31255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:39.039884 31255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:39.050225 31260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:39.050287 31263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:39.050262 31266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:39.051322 31255 server_base.cc:1034] running on GCE node
I20260809 15:06:39.051928 31255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:39.053042 31255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:39.054235 31255 hybrid_clock.cc:648] HybridClock initialized: now 1786287999054200 us; error 30 us; skew 500 ppm
I20260809 15:06:39.054526 31255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:39 dist-test-slave-q9h9 krb5kdc[31228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786287999, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:39.057317 31255 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:39.058640 31255 webserver.cc:460] Webserver started at http://127.27.186.190:34993/ using document root <none> and password file <none>
I20260809 15:06:39.059233 31255 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:39.059346 31255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:39.059607 31255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:39.062294 31255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "e9b4d54349c84f71a22213e34a12afa3"
format_stamp: "Formatted at 2026-08-09 15:06:39 on dist-test-slave-q9h9"
server_key: "11ef3c24b521ddca5a63924324167cfd"
server_key_iv: "2edb83bb8522cac6597b58a325dbc484"
server_key_version: "encryptionkey@0"
I20260809 15:06:39.062973 31255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "e9b4d54349c84f71a22213e34a12afa3"
format_stamp: "Formatted at 2026-08-09 15:06:39 on dist-test-slave-q9h9"
server_key: "11ef3c24b521ddca5a63924324167cfd"
server_key_iv: "2edb83bb8522cac6597b58a325dbc484"
server_key_version: "encryptionkey@0"
I20260809 15:06:39.067503 31255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:06:39.070621 31279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:39.071973 31255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:06:39.072129 31255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "e9b4d54349c84f71a22213e34a12afa3"
format_stamp: "Formatted at 2026-08-09 15:06:39 on dist-test-slave-q9h9"
server_key: "11ef3c24b521ddca5a63924324167cfd"
server_key_iv: "2edb83bb8522cac6597b58a325dbc484"
server_key_version: "encryptionkey@0"
I20260809 15:06:39.072294 31255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:39.094985 31255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:39.098368 31255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:39.098575 31255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:39.115653 31255 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:42141
I20260809 15:06:39.115686 31357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:42141 every 8 connection(s)
I20260809 15:06:39.116801 31255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:39.120071 31358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:39.123827 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31255
I20260809 15:06:39.123976 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:39.124377 28394 external_mini_cluster.cc:1428] Setting key 3bc5160e9f0bf7e07049b8690e3c56d7
I20260809 15:06:39.131485 31358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3: Bootstrap starting.
I20260809 15:06:39.134781 31358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:39.136020 31358 log.cc:824] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:39.138926 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:39.125931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45143 (local address 127.27.186.190:42141)
0809 15:06:39.126714 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:39.126729 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:39.126788 (+ 59us) server_negotiation.cc:407] Connection header received
0809 15:06:39.129650 (+ 2862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:39.129677 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:39.130149 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:39.130532 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:39.131004 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:39.132927 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:39.134306 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:39.135321 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:39.135488 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:39.136534 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:39.136544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:39.136560 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:39.137114 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:39.137197 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:39.137226 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:39.137385 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:39.137604 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 15:06:39.137757 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":283,"threads_started":1}
I20260809 15:06:39.139004 31358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3: No bootstrap required, opened a new log
I20260809 15:06:39.142304 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:39.146447 31358 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b4d54349c84f71a22213e34a12afa3" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42141 } }
I20260809 15:06:39.146797 31358 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:39.146889 31358 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b4d54349c84f71a22213e34a12afa3, State: Initialized, Role: FOLLOWER
I20260809 15:06:39.147483 31358 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b4d54349c84f71a22213e34a12afa3" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42141 } }
I20260809 15:06:39.147709 31358 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:39.147872 31358 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:39.148037 31358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:39.150134 31358 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b4d54349c84f71a22213e34a12afa3" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42141 } }
I20260809 15:06:39.150656 31358 leader_election.cc:302] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b4d54349c84f71a22213e34a12afa3; no voters:
I20260809 15:06:39.151288 31358 leader_election.cc:288] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:39.151382 31380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:39.152531 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:39.146373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46023 (local address 127.27.186.190:42141)
0809 15:06:39.146533 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:39.146538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:39.146906 (+ 368us) server_negotiation.cc:407] Connection header received
0809 15:06:39.147052 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:39.147057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:39.147122 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:39.147211 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:39.147725 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:39.148539 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:39.149787 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:39.150288 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:39.150406 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:39.151579 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:39.151585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:39.151593 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:39.151613 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:39.151687 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:39.151712 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:39.151816 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:39.152146 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 15:06:39.152198 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:06:39.153458 31358 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:39.153127 31380 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [term 1 LEADER]: Becoming Leader. State: Replica: e9b4d54349c84f71a22213e34a12afa3, State: Running, Role: LEADER
I20260809 15:06:39.154498 31380 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b4d54349c84f71a22213e34a12afa3" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42141 } }
I20260809 15:06:39.157451 31381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b4d54349c84f71a22213e34a12afa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b4d54349c84f71a22213e34a12afa3" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42141 } } }
I20260809 15:06:39.163687 31381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:39.163753 31382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b4d54349c84f71a22213e34a12afa3. Latest consensus state: current_term: 1 leader_uuid: "e9b4d54349c84f71a22213e34a12afa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b4d54349c84f71a22213e34a12afa3" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 42141 } } }
I20260809 15:06:39.164386 31382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:39.166514 31400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:39.170629 31400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:39.177879 31400 catalog_manager.cc:1347] Generated new cluster ID: 9809f8afb4ec44cc8af5bba4c8fb712a
I20260809 15:06:39.177997 31400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:39.196935 31400 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:39.198632 31400 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:39.207155 31400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b4d54349c84f71a22213e34a12afa3: Generated new TSK 0
I20260809 15:06:39.208024 31400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:39.215365 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31255
W20260809 15:06:39.216385 31351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45143
W20260809 15:06:39.217061 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31255: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:39.217214 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31255
W20260809 15:06:39.237599 31224 connection.cc:535] client connection to 127.27.186.190:42141 recv error: Network error: failed to read from TLS socket (remote: 127.27.186.190:42141): Connection reset by peer (error 104)
2026-08-09T15:06:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:39 dist-test-slave-q9h9 krb5kdc[31409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:39 dist-test-slave-q9h9 krb5kdc[31409](info): set up 2 sockets
Aug 09 15:06:39 dist-test-slave-q9h9 krb5kdc[31409](info): commencing operation
krb5kdc: starting...
W20260809 15:06:40.424093 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:40 dist-test-slave-q9h9 krb5kdc[31409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:40.658895 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38785
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:38785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:40.965844 31430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:40.966127 31430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:40.966259 31430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:40.973086 31430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:40.973193 31430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:40.973269 31430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:40.973351 31430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:40.973408 31430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:40.982502 31430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:38785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:38785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:40.983532 31430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:40.984702 31430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:40.995335 31438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:40.995369 31440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:40.995275 31448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:40.996124 31430 server_base.cc:1034] running on GCE node
I20260809 15:06:40.996815 31430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:40.998035 31430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:40.999233 31430 hybrid_clock.cc:648] HybridClock initialized: now 1786288000999189 us; error 38 us; skew 500 ppm
I20260809 15:06:40.999545 31430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:41 dist-test-slave-q9h9 krb5kdc[31409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288001, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:41.005752 31430 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:41.007112 31430 webserver.cc:460] Webserver started at http://127.27.186.190:42729/ using document root <none> and password file <none>
I20260809 15:06:41.007694 31430 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:41.007817 31430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:41.008100 31430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:41.010833 31430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "7ac1caaae6a946c49d89736d28b6c29d"
format_stamp: "Formatted at 2026-08-09 15:06:41 on dist-test-slave-q9h9"
server_key: "87da17ebd4fb989e161f7f3d803c5673"
server_key_iv: "a834cb002277e536515c1d7f9c41d3e7"
server_key_version: "encryptionkey@0"
I20260809 15:06:41.011547 31430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "7ac1caaae6a946c49d89736d28b6c29d"
format_stamp: "Formatted at 2026-08-09 15:06:41 on dist-test-slave-q9h9"
server_key: "87da17ebd4fb989e161f7f3d803c5673"
server_key_iv: "a834cb002277e536515c1d7f9c41d3e7"
server_key_version: "encryptionkey@0"
I20260809 15:06:41.016283 31430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:06:41.019506 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:41.020638 31430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 15:06:41.020769 31430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "7ac1caaae6a946c49d89736d28b6c29d"
format_stamp: "Formatted at 2026-08-09 15:06:41 on dist-test-slave-q9h9"
server_key: "87da17ebd4fb989e161f7f3d803c5673"
server_key_iv: "a834cb002277e536515c1d7f9c41d3e7"
server_key_version: "encryptionkey@0"
I20260809 15:06:41.020926 31430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:41.025951 31430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:41.029692 31430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:41.029923 31430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:41.046445 31430 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:38785
I20260809 15:06:41.046496 31532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:38785 every 8 connection(s)
I20260809 15:06:41.047623 31430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:41.049273 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31430
I20260809 15:06:41.049422 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:41.049861 28394 external_mini_cluster.cc:1428] Setting key adf03dc1fed1b2b43c355517aa167c59
I20260809 15:06:41.051229 31533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:41.056767 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:41.051447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52097 (local address 127.27.186.190:38785)
0809 15:06:41.052173 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:41.052183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:41.052222 (+ 39us) server_negotiation.cc:407] Connection header received
0809 15:06:41.053716 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:41.053735 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:41.054067 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:41.054395 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:41.055404 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:41.055410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:41.055421 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:41.055435 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:41.055505 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:41.055525 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:41.055687 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:41.055783 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 15:06:41.055934 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":335,"threads_started":1}
I20260809 15:06:41.060156 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:41.064007 31533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d: Bootstrap starting.
I20260809 15:06:41.067339 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:41.065198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54819 (local address 127.27.186.190:38785)
0809 15:06:41.065324 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:41.065328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:41.065713 (+ 385us) server_negotiation.cc:407] Connection header received
0809 15:06:41.065861 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:41.065865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:41.065903 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:41.065980 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:41.066818 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:41.066822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:41.066825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:41.066837 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:41.066870 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:41.066883 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:41.066959 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:41.067070 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 15:06:41.067107 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 15:06:41.067526 31533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:41.068634 31533 log.cc:824] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:41.071043 31533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d: No bootstrap required, opened a new log
I20260809 15:06:41.077765 31533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac1caaae6a946c49d89736d28b6c29d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38785 } }
I20260809 15:06:41.078066 31533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:41.078158 31533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac1caaae6a946c49d89736d28b6c29d, State: Initialized, Role: FOLLOWER
I20260809 15:06:41.078630 31533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac1caaae6a946c49d89736d28b6c29d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38785 } }
I20260809 15:06:41.078825 31533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:41.078917 31533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:41.079036 31533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:41.080920 31533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac1caaae6a946c49d89736d28b6c29d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38785 } }
I20260809 15:06:41.081324 31533 leader_election.cc:302] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ac1caaae6a946c49d89736d28b6c29d; no voters:
I20260809 15:06:41.081808 31533 leader_election.cc:288] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:41.081923 31555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:41.082576 31555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [term 1 LEADER]: Becoming Leader. State: Replica: 7ac1caaae6a946c49d89736d28b6c29d, State: Running, Role: LEADER
I20260809 15:06:41.083052 31555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac1caaae6a946c49d89736d28b6c29d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38785 } }
I20260809 15:06:41.083534 31533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:41.085600 31556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ac1caaae6a946c49d89736d28b6c29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac1caaae6a946c49d89736d28b6c29d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38785 } } }
I20260809 15:06:41.086074 31556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:41.085577 31557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ac1caaae6a946c49d89736d28b6c29d. Latest consensus state: current_term: 1 leader_uuid: "7ac1caaae6a946c49d89736d28b6c29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac1caaae6a946c49d89736d28b6c29d" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 38785 } } }
I20260809 15:06:41.087921 31557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:41.089469 31562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:41.094650 31562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:41.102895 31562 catalog_manager.cc:1347] Generated new cluster ID: f0d027edcfbf47a1b53a8ff905f8e8f5
I20260809 15:06:41.102981 31562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:41.121472 31562 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:41.121804 31562 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:41.128770 31562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ac1caaae6a946c49d89736d28b6c29d: Generated new TSK 0
I20260809 15:06:41.129390 31562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:41.144735 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31430
W20260809 15:06:41.145828 31525 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52097
W20260809 15:06:41.146492 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:41.146615 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31430
2026-08-09T15:06:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:41 dist-test-slave-q9h9 krb5kdc[31591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:41 dist-test-slave-q9h9 krb5kdc[31591](info): set up 2 sockets
Aug 09 15:06:41 dist-test-slave-q9h9 krb5kdc[31591](info): commencing operation
krb5kdc: starting...
W20260809 15:06:42.360549 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:42 dist-test-slave-q9h9 krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:42.600361 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40811
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:40811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:42.904032 31617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:42.904264 31617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:42.904366 31617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:42.911083 31617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:42.911175 31617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:42.911226 31617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:42.911293 31617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:42.911336 31617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:42.920377 31617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:40811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:42.921327 31617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:42.922439 31617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:42.932926 31630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:42.932942 31627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:42.932965 31628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:42.933756 31617 server_base.cc:1034] running on GCE node
I20260809 15:06:42.934504 31617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:42.935737 31617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:42.936928 31617 hybrid_clock.cc:648] HybridClock initialized: now 1786288002936884 us; error 41 us; skew 500 ppm
I20260809 15:06:42.937228 31617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:42 dist-test-slave-q9h9 krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288002, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:42.940053 31617 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:42.941320 31617 webserver.cc:460] Webserver started at http://127.27.186.190:34383/ using document root <none> and password file <none>
I20260809 15:06:42.941885 31617 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:42.941983 31617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:42.942258 31617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:42.944855 31617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "eb3001310db14e8ca2824499a97f098c"
format_stamp: "Formatted at 2026-08-09 15:06:42 on dist-test-slave-q9h9"
server_key: "f37d652fe5d62a3f5320b4aa12fb027f"
server_key_iv: "890a6f11a305f8d23e4f1e9e5c29cfed"
server_key_version: "encryptionkey@0"
I20260809 15:06:42.945516 31617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "eb3001310db14e8ca2824499a97f098c"
format_stamp: "Formatted at 2026-08-09 15:06:42 on dist-test-slave-q9h9"
server_key: "f37d652fe5d62a3f5320b4aa12fb027f"
server_key_iv: "890a6f11a305f8d23e4f1e9e5c29cfed"
server_key_version: "encryptionkey@0"
I20260809 15:06:42.950103 31617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260809 15:06:42.953217 31646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:42.954420 31617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:42.954562 31617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "eb3001310db14e8ca2824499a97f098c"
format_stamp: "Formatted at 2026-08-09 15:06:42 on dist-test-slave-q9h9"
server_key: "f37d652fe5d62a3f5320b4aa12fb027f"
server_key_iv: "890a6f11a305f8d23e4f1e9e5c29cfed"
server_key_version: "encryptionkey@0"
I20260809 15:06:42.954723 31617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:42.986444 31617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:42.989840 31617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:42.990097 31617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:43.006975 31617 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:40811
I20260809 15:06:43.007004 31745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:40811 every 8 connection(s)
I20260809 15:06:43.008127 31617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:43.009471 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31617
I20260809 15:06:43.009652 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:43.010236 28394 external_mini_cluster.cc:1428] Setting key d9574f05cffc0015790a9e8038d12855
I20260809 15:06:43.012267 31747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 15:06:43 dist-test-slave-q9h9 krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:43.024466 31747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c: Bootstrap starting.
I20260809 15:06:43.028239 31747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:43.029448 31747 log.cc:824] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:43.032495 31747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c: No bootstrap required, opened a new log
I20260809 15:06:43.032754 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:43.011839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59811 (local address 127.27.186.190:40811)
0809 15:06:43.012591 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:43.012602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:43.012639 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:06:43.014258 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:43.014284 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:43.014628 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:43.014941 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:43.016637 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:43.017930 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:43.019652 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:43.020374 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:43.020505 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:43.024983 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:43.024993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:43.025010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:43.025050 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:43.028760 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:43.029409 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:43.029418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:43.029426 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:43.029626 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:43.029992 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:43.029998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:43.030002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:43.030532 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:43.030795 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:43.031166 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 15:06:43.031559 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":338,"threads_started":1}
I20260809 15:06:43.036083 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:43.041532 31747 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3001310db14e8ca2824499a97f098c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40811 } }
I20260809 15:06:43.041949 31747 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:43.042104 31747 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3001310db14e8ca2824499a97f098c, State: Initialized, Role: FOLLOWER
I20260809 15:06:43.042618 31747 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3001310db14e8ca2824499a97f098c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40811 } }
I20260809 15:06:43.042824 31747 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:43.042948 31747 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:43.043095 31747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:43.045624 31747 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3001310db14e8ca2824499a97f098c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40811 } }
I20260809 15:06:43.046236 31747 leader_election.cc:302] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3001310db14e8ca2824499a97f098c; no voters:
I20260809 15:06:43.046844 31747 leader_election.cc:288] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:43.046938 31763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:43.048882 31763 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [term 1 LEADER]: Becoming Leader. State: Replica: eb3001310db14e8ca2824499a97f098c, State: Running, Role: LEADER
I20260809 15:06:43.048904 31747 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:43.049448 31763 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3001310db14e8ca2824499a97f098c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40811 } }
I20260809 15:06:43.052191 31765 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3001310db14e8ca2824499a97f098c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3001310db14e8ca2824499a97f098c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40811 } } }
I20260809 15:06:43.053515 31765 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:43.052771 31767 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3001310db14e8ca2824499a97f098c. Latest consensus state: current_term: 1 leader_uuid: "eb3001310db14e8ca2824499a97f098c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3001310db14e8ca2824499a97f098c" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 40811 } } }
I20260809 15:06:43.054477 31772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:43.054526 31767 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:43.056005 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:43.040983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60983 (local address 127.27.186.190:40811)
0809 15:06:43.041138 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:43.041143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:43.041584 (+ 441us) server_negotiation.cc:407] Connection header received
0809 15:06:43.041688 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:43.041692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:43.041762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:43.041864 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:43.043376 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:43.044459 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:43.045573 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:43.046236 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:43.046359 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:43.048689 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:43.048716 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:43.048721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:43.048758 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:43.052202 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:43.052999 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:43.053005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:43.053008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:43.053138 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:43.053522 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:43.053527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:43.053530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:43.053899 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:43.055180 (+ 1281us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:43.055381 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 15:06:43.055597 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 15:06:43.067970 31772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:43.075910 31772 catalog_manager.cc:1347] Generated new cluster ID: b84bdc17c46a490f8f7e73f83caca0db
I20260809 15:06:43.076013 31772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:43.100955 31772 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:43.102113 31772 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:43.108536 31772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3001310db14e8ca2824499a97f098c: Generated new TSK 0
I20260809 15:06:43.109253 31772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:43.113835 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31617
I20260809 15:06:43.115116 31739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:43.200052 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31617
2026-08-09T15:06:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2050 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:43 dist-test-slave-q9h9 krb5kdc[31802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:43 dist-test-slave-q9h9 krb5kdc[31802](info): set up 2 sockets
Aug 09 15:06:43 dist-test-slave-q9h9 krb5kdc[31802](info): commencing operation
krb5kdc: starting...
W20260809 15:06:44.412358 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:44 dist-test-slave-q9h9 krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:44.662678 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37951
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:37951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:44.967722 31829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:44.967958 31829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:44.968050 31829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:44.974818 31829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:44.974917 31829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:44.974968 31829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:44.975024 31829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:44.975065 31829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:44.984081 31829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:37951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:44.985056 31829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:44.986172 31829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:44.996718 31836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:44.996748 31839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:44.996781 31837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:44.997466 31829 server_base.cc:1034] running on GCE node
I20260809 15:06:44.998167 31829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:44.999331 31829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:45.000511 31829 hybrid_clock.cc:648] HybridClock initialized: now 1786288005000466 us; error 42 us; skew 500 ppm
I20260809 15:06:45.000792 31829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:45 dist-test-slave-q9h9 krb5kdc[31802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288005, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:45.003643 31829 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:45.004880 31829 webserver.cc:460] Webserver started at http://127.27.186.190:33135/ using document root <none> and password file <none>
I20260809 15:06:45.005453 31829 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:45.005553 31829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:45.005791 31829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:45.008461 31829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "ed67873ccb3e42268a61e442cef9c1df"
format_stamp: "Formatted at 2026-08-09 15:06:45 on dist-test-slave-q9h9"
server_key: "bbc413c53217b31c85229611aefaf18b"
server_key_iv: "1889518d5314bacfcce01bcb2dc0714e"
server_key_version: "encryptionkey@0"
I20260809 15:06:45.009172 31829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "ed67873ccb3e42268a61e442cef9c1df"
format_stamp: "Formatted at 2026-08-09 15:06:45 on dist-test-slave-q9h9"
server_key: "bbc413c53217b31c85229611aefaf18b"
server_key_iv: "1889518d5314bacfcce01bcb2dc0714e"
server_key_version: "encryptionkey@0"
I20260809 15:06:45.013790 31829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260809 15:06:45.016891 31852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:45.018087 31829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:45.018214 31829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "ed67873ccb3e42268a61e442cef9c1df"
format_stamp: "Formatted at 2026-08-09 15:06:45 on dist-test-slave-q9h9"
server_key: "bbc413c53217b31c85229611aefaf18b"
server_key_iv: "1889518d5314bacfcce01bcb2dc0714e"
server_key_version: "encryptionkey@0"
I20260809 15:06:45.018358 31829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:45.032320 31829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:45.035748 31829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:45.035969 31829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:45.052464 31829 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:37951
I20260809 15:06:45.052495 31944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:37951 every 8 connection(s)
I20260809 15:06:45.053580 31829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:45.056857 31949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:45.061444 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 31829
I20260809 15:06:45.061622 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:45.062170 28394 external_mini_cluster.cc:1428] Setting key 91ee39ef183d9936af08bc3b84d0dba1
I20260809 15:06:45.067312 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df: Bootstrap starting.
I20260809 15:06:45.070538 31949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:45.071666 31949 log.cc:824] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:45.074460 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df: No bootstrap required, opened a new log
I20260809 15:06:45.074522 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:45.063724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40301 (local address 127.27.186.190:37951)
0809 15:06:45.064427 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:45.064438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:45.064474 (+ 36us) server_negotiation.cc:407] Connection header received
0809 15:06:45.066199 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:45.066220 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:45.066546 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:45.066841 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:45.067399 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:45.068634 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:45.070435 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:45.071088 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:45.071211 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:45.072396 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:45.072407 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:45.072423 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:45.073044 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:45.073109 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:45.073137 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:45.073218 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:45.073428 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 15:06:45.073628 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":318,"threads_started":1}
I20260809 15:06:45.077855 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:45.082083 31949 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed67873ccb3e42268a61e442cef9c1df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37951 } }
I20260809 15:06:45.082515 31949 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:45.082633 31949 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed67873ccb3e42268a61e442cef9c1df, State: Initialized, Role: FOLLOWER
I20260809 15:06:45.083249 31949 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed67873ccb3e42268a61e442cef9c1df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37951 } }
I20260809 15:06:45.083516 31949 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:45.083658 31949 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:45.083818 31949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:45.086650 31949 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed67873ccb3e42268a61e442cef9c1df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37951 } }
I20260809 15:06:45.087201 31949 leader_election.cc:302] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed67873ccb3e42268a61e442cef9c1df; no voters:
I20260809 15:06:45.087846 31949 leader_election.cc:288] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:45.087937 31964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:45.088238 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:45.081835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36919 (local address 127.27.186.190:37951)
0809 15:06:45.081991 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:45.081997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:45.082427 (+ 430us) server_negotiation.cc:407] Connection header received
0809 15:06:45.082608 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:45.082613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:45.082674 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:45.082758 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:45.083353 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:45.084155 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:45.085902 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:45.086400 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:45.086488 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:45.087479 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:45.087484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:45.087487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:45.087501 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:45.087534 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:45.087548 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:45.087633 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:45.087803 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 15:06:45.087888 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:06:45.089195 31964 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [term 1 LEADER]: Becoming Leader. State: Replica: ed67873ccb3e42268a61e442cef9c1df, State: Running, Role: LEADER
I20260809 15:06:45.089773 31964 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed67873ccb3e42268a61e442cef9c1df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37951 } }
I20260809 15:06:45.089972 31949 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:45.092236 31965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed67873ccb3e42268a61e442cef9c1df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed67873ccb3e42268a61e442cef9c1df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37951 } } }
I20260809 15:06:45.092587 31965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:45.093179 31966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed67873ccb3e42268a61e442cef9c1df. Latest consensus state: current_term: 1 leader_uuid: "ed67873ccb3e42268a61e442cef9c1df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed67873ccb3e42268a61e442cef9c1df" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 37951 } } }
I20260809 15:06:45.093497 31966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:45.093598 31977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:45.105809 31977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:45.114177 31977 catalog_manager.cc:1347] Generated new cluster ID: de69456511334ab689a21cdc02826e27
I20260809 15:06:45.114271 31977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:45.172820 31977 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:45.174422 31977 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:45.180714 31977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed67873ccb3e42268a61e442cef9c1df: Generated new TSK 0
I20260809 15:06:45.181591 31977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:45.189168 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31829
W20260809 15:06:45.190140 31935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40301
W20260809 15:06:45.190822 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 31829: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:45.190943 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 31829
2026-08-09T15:06:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:45 dist-test-slave-q9h9 krb5kdc[32002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:45 dist-test-slave-q9h9 krb5kdc[32002](info): set up 2 sockets
Aug 09 15:06:45 dist-test-slave-q9h9 krb5kdc[32002](info): commencing operation
krb5kdc: starting...
W20260809 15:06:46.401181 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:46 dist-test-slave-q9h9 krb5kdc[32002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:46.645979 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:45263
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:46.954545 32023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:46.954810 32023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:46.954964 32023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:46.961798 32023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:46.961907 32023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:46.961987 32023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:46.962091 32023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:46.962152 32023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:46.971366 32023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:46.972468 32023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:46.973680 32023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:46.984911 32036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:46.985000 32040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:46.984957 32037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:46.985942 32023 server_base.cc:1034] running on GCE node
I20260809 15:06:46.986662 32023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:46.987885 32023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:46.989087 32023 hybrid_clock.cc:648] HybridClock initialized: now 1786288006989044 us; error 30 us; skew 500 ppm
I20260809 15:06:46.989405 32023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:46 dist-test-slave-q9h9 krb5kdc[32002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:46.996886 32023 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:46.998253 32023 webserver.cc:460] Webserver started at http://127.27.186.190:37825/ using document root <none> and password file <none>
I20260809 15:06:46.998858 32023 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:46.998971 32023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:46.999251 32023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:47.001973 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "6fb6d6d5593a4e82932c885256b004d9"
format_stamp: "Formatted at 2026-08-09 15:06:46 on dist-test-slave-q9h9"
server_key: "f1fe302aef26a976d8b9f0e3215fafd0"
server_key_iv: "6c4d55cecaac6ff064c079cb96b1c027"
server_key_version: "encryptionkey@0"
I20260809 15:06:47.002717 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "6fb6d6d5593a4e82932c885256b004d9"
format_stamp: "Formatted at 2026-08-09 15:06:46 on dist-test-slave-q9h9"
server_key: "f1fe302aef26a976d8b9f0e3215fafd0"
server_key_iv: "6c4d55cecaac6ff064c079cb96b1c027"
server_key_version: "encryptionkey@0"
I20260809 15:06:47.007403 32023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:06:47.010586 32055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:47.011755 32023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:47.011900 32023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "6fb6d6d5593a4e82932c885256b004d9"
format_stamp: "Formatted at 2026-08-09 15:06:46 on dist-test-slave-q9h9"
server_key: "f1fe302aef26a976d8b9f0e3215fafd0"
server_key_iv: "6c4d55cecaac6ff064c079cb96b1c027"
server_key_version: "encryptionkey@0"
I20260809 15:06:47.012063 32023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:47.017546 32023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:47.021323 32023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:47.021586 32023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:47.038827 32023 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:45263
I20260809 15:06:47.038878 32144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:45263 every 8 connection(s)
I20260809 15:06:47.040026 32023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:47.043301 32146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:47.046418 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 32023
I20260809 15:06:47.046590 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:47.047151 28394 external_mini_cluster.cc:1428] Setting key dbd41a00c50c835cf293dac90b7585fa
I20260809 15:06:47.054071 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:47.048758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37583 (local address 127.27.186.190:45263)
0809 15:06:47.049422 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:47.049432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:47.049470 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:06:47.051096 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:47.051119 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:47.051425 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:47.051743 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:47.052682 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:47.052687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:47.052699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:47.052712 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:47.052774 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:47.052793 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:47.052956 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:47.053049 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 15:06:47.053195 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":298,"threads_started":1}
I20260809 15:06:47.054797 32146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9: Bootstrap starting.
I20260809 15:06:47.057596 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:47.058202 32146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:47.059381 32146 log.cc:824] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:47.062317 32146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9: No bootstrap required, opened a new log
I20260809 15:06:47.064592 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:47.062338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41371 (local address 127.27.186.190:45263)
0809 15:06:47.062502 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:47.062508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:47.062955 (+ 447us) server_negotiation.cc:407] Connection header received
0809 15:06:47.063099 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:47.063102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:47.063142 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:47.063214 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:47.064045 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:47.064049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:47.064052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 15:06:47.064064 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:47.064097 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 15:06:47.064113 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:47.064201 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:47.064301 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 15:06:47.064343 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 15:06:47.070134 32146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb6d6d5593a4e82932c885256b004d9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45263 } }
I20260809 15:06:47.070472 32146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:06:47.070557 32146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb6d6d5593a4e82932c885256b004d9, State: Initialized, Role: FOLLOWER
I20260809 15:06:47.071036 32146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb6d6d5593a4e82932c885256b004d9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45263 } }
I20260809 15:06:47.071259 32146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:06:47.071382 32146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:06:47.071525 32146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:47.073485 32146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb6d6d5593a4e82932c885256b004d9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45263 } }
I20260809 15:06:47.073915 32146 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb6d6d5593a4e82932c885256b004d9; no voters:
I20260809 15:06:47.074474 32146 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:06:47.074579 32160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:47.075693 32160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb6d6d5593a4e82932c885256b004d9, State: Running, Role: LEADER
I20260809 15:06:47.076251 32160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb6d6d5593a4e82932c885256b004d9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45263 } }
I20260809 15:06:47.076376 32146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:47.078797 32161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb6d6d5593a4e82932c885256b004d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb6d6d5593a4e82932c885256b004d9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45263 } } }
I20260809 15:06:47.078769 32162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb6d6d5593a4e82932c885256b004d9. Latest consensus state: current_term: 1 leader_uuid: "6fb6d6d5593a4e82932c885256b004d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb6d6d5593a4e82932c885256b004d9" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 45263 } } }
I20260809 15:06:47.079219 32162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:47.079218 32161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:47.082942 32170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:47.087316 32170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:47.094311 32170 catalog_manager.cc:1347] Generated new cluster ID: f92dc8d0d0a74051a33d76c997ab8c02
I20260809 15:06:47.094408 32170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:47.119174 32170 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:47.119597 32170 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:47.130180 32170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb6d6d5593a4e82932c885256b004d9: Generated new TSK 0
I20260809 15:06:47.131006 32170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:47.137550 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32023
W20260809 15:06:47.138620 32139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37583
W20260809 15:06:47.139283 28394 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32023: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 15:06:47.139428 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 32023
2026-08-09T15:06:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1949 ms)
[----------] 18 tests from AuthTokenIssuingTest (35838 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:47 dist-test-slave-q9h9 krb5kdc[32197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:47 dist-test-slave-q9h9 krb5kdc[32197](info): set up 2 sockets
Aug 09 15:06:47 dist-test-slave-q9h9 krb5kdc[32197](info): commencing operation
krb5kdc: starting...
W20260809 15:06:48.365783 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.004s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:48 dist-test-slave-q9h9 krb5kdc[32197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:48.609202 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:35125
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:35125,127.27.186.189:37051,127.27.186.188:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:48.918793 32215 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 15:06:48.919070 32215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:48.919168 32215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:48.919286 32215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:48.926172 32215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:48.926244 32215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:48.926337 32215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:48.926397 32215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:48.926481 32215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:48.935742 32215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:35125,127.27.186.189:37051,127.27.186.188:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:48.936784 32215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:48.937927 32215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:48.948618 32230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:48.948705 32227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:48.948675 32225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:48.949505 32215 server_base.cc:1034] running on GCE node
I20260809 15:06:48.950212 32215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:48.951382 32215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:48.952595 32215 hybrid_clock.cc:648] HybridClock initialized: now 1786288008952531 us; error 41 us; skew 500 ppm
I20260809 15:06:48.952903 32215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:48 dist-test-slave-q9h9 krb5kdc[32197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:48.955854 32215 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:48.957118 32215 webserver.cc:460] Webserver started at http://127.27.186.190:37111/ using document root <none> and password file <none>
I20260809 15:06:48.957737 32215 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:48.957844 32215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:48.958221 32215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:48.960901 32215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "b6d871e9721e4638ad1233faeb5a3e1f"
format_stamp: "Formatted at 2026-08-09 15:06:48 on dist-test-slave-q9h9"
server_key: "b9fb0f5c4f80f57cc3ef4759fc9d66d6"
server_key_iv: "7fb654513926ba590ec97fd78bb7c433"
server_key_version: "encryptionkey@0"
I20260809 15:06:48.961648 32215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "b6d871e9721e4638ad1233faeb5a3e1f"
format_stamp: "Formatted at 2026-08-09 15:06:48 on dist-test-slave-q9h9"
server_key: "b9fb0f5c4f80f57cc3ef4759fc9d66d6"
server_key_iv: "7fb654513926ba590ec97fd78bb7c433"
server_key_version: "encryptionkey@0"
I20260809 15:06:48.966452 32215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 15:06:48.969640 32240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:48.970841 32215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:48.970999 32215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "b6d871e9721e4638ad1233faeb5a3e1f"
format_stamp: "Formatted at 2026-08-09 15:06:48 on dist-test-slave-q9h9"
server_key: "b9fb0f5c4f80f57cc3ef4759fc9d66d6"
server_key_iv: "7fb654513926ba590ec97fd78bb7c433"
server_key_version: "encryptionkey@0"
I20260809 15:06:48.971192 32215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:48.991163 32215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:48.994935 32215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:48.995160 32215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:49.012145 32215 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:35125
I20260809 15:06:49.012178 32318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:35125 every 8 connection(s)
I20260809 15:06:49.013262 32215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:49.016510 32321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:49.019256 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 32215
I20260809 15:06:49.019464 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:49.020081 28394 external_mini_cluster.cc:1428] Setting key 93d1257665aadf56e9c56d73d6b74cfc
I20260809 15:06:49.022567 32321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:49.043056 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.025257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:35125 (local address 127.0.0.1:58620)
0809 15:06:49.026289 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.026328 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:49.027949 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:49.028684 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:49.028690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:49.029406 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:49.029743 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.029760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.031304 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.031308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.032329 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.032342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.033227 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.033232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.033378 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.033922 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:49.033944 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:49.036017 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:49.039635 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.039645 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.039652 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.040029 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.040470 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.040473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.040476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.040616 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.041403 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:49.041407 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:49.041584 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 15:06:49.041781 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 15:06:49.042145 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":647,"threads_started":1}
I20260809 15:06:49.043109 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.025254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.27.186.190:35125)
0809 15:06:49.025978 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:49.025992 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:49.026448 (+ 456us) server_negotiation.cc:407] Connection header received
0809 15:06:49.028285 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:49.028290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:49.028482 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:49.028653 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:49.029933 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.031161 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.032462 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.033002 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.033195 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.036236 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:49.036246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:49.036258 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:49.036296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:49.039509 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.040158 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.040166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.040169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.040364 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.040718 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.040723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.040727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.041279 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:49.041506 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:49.041910 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 15:06:49.042311 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":509,"thread_start_us":327,"threads_started":1}
I20260809 15:06:49.045678 32321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:49.046605 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.046258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40746)
0809 15:06:49.046415 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.046505 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 15:06:49.046868 32245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
I20260809 15:06:49.048238 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.047951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40760)
0809 15:06:49.048064 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.048138 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 15:06:49.048444 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:37051: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.186.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab.
Entry for principal kudu/127.27.186.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab.
I20260809 15:06:49.080864 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:37051
--webserver_interface=127.27.186.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab
--principal=kudu/127.27.186.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:35125,127.27.186.189:37051,127.27.186.188:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:06:49.097730 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } attempt: 1
I20260809 15:06:49.098690 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.098306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40764)
0809 15:06:49.098468 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.098561 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 15:06:49.099561 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.099230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40768)
0809 15:06:49.099376 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.099440 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 15:06:49.099722 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:37051: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
I20260809 15:06:49.167951 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } attempt: 2
I20260809 15:06:49.169063 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.168606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40772)
0809 15:06:49.168830 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.168927 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 15:06:49.170001 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.169650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40780)
0809 15:06:49.169791 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.169873 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 15:06:49.170202 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:37051: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
I20260809 15:06:49.261477 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } attempt: 3
I20260809 15:06:49.262804 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.262303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40784)
0809 15:06:49.262530 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.262637 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 15:06:49.264026 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.263619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40786)
0809 15:06:49.263786 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.263877 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 15:06:49.264266 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:37051: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
W20260809 15:06:49.400269 32335 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 15:06:49.400563 32335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:49.400658 32335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:49.400771 32335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 15:06:49.407490 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } attempt: 4
W20260809 15:06:49.407644 32335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:49.407734 32335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:49.407821 32335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:49.407894 32335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:49.407977 32335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:49.408664 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.408089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40802)
0809 15:06:49.408398 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.408504 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260809 15:06:49.409655 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.409275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40814)
0809 15:06:49.409413 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.409499 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 15:06:49.409875 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:37051: Network error: Client connection negotiation failed: client connection to 127.27.186.189:37051: connect: Connection refused (error 111)
I20260809 15:06:49.417691 32335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:35125,127.27.186.189:37051,127.27.186.188:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab
--principal=kudu/127.27.186.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.186.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:49.418744 32335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:49.419853 32335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:49.430487 32347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:49.430474 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:49.431020 32342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:49.431586 32335 server_base.cc:1034] running on GCE node
I20260809 15:06:49.432262 32335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:49.433442 32335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:49.434646 32335 hybrid_clock.cc:648] HybridClock initialized: now 1786288009434580 us; error 42 us; skew 500 ppm
I20260809 15:06:49.434949 32335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:49.437866 32335 init.cc:377] Logged in from keytab as kudu/127.27.186.189@KRBTEST.COM (short username kudu)
I20260809 15:06:49.439133 32335 webserver.cc:460] Webserver started at http://127.27.186.189:40231/ using document root <none> and password file <none>
I20260809 15:06:49.439745 32335 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:49.439847 32335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:49.440127 32335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:49.442862 32335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data/instance:
uuid: "aaedf61362c74b65a49a88825eff2000"
format_stamp: "Formatted at 2026-08-09 15:06:49 on dist-test-slave-q9h9"
server_key: "616b1e1ebaef439d8a1ad1c1c53b53a5"
server_key_iv: "9857241ec1a8409590fcf0f8bf317e45"
server_key_version: "encryptionkey@0"
I20260809 15:06:49.443583 32335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal/instance:
uuid: "aaedf61362c74b65a49a88825eff2000"
format_stamp: "Formatted at 2026-08-09 15:06:49 on dist-test-slave-q9h9"
server_key: "616b1e1ebaef439d8a1ad1c1c53b53a5"
server_key_iv: "9857241ec1a8409590fcf0f8bf317e45"
server_key_version: "encryptionkey@0"
I20260809 15:06:49.448185 32335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:06:49.451297 32354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:49.452430 32335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:49.452584 32335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal
uuid: "aaedf61362c74b65a49a88825eff2000"
format_stamp: "Formatted at 2026-08-09 15:06:49 on dist-test-slave-q9h9"
server_key: "616b1e1ebaef439d8a1ad1c1c53b53a5"
server_key_iv: "9857241ec1a8409590fcf0f8bf317e45"
server_key_version: "encryptionkey@0"
I20260809 15:06:49.452764 32335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:49.462445 32335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:49.466167 32335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:49.466401 32335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:49.481993 32335 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.189:37051
I20260809 15:06:49.482050 32427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.189:37051 every 8 connection(s)
I20260809 15:06:49.483141 32335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
I20260809 15:06:49.486335 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:49.492003 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 32335
I20260809 15:06:49.492165 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-1/wal/instance
I20260809 15:06:49.492571 28394 external_mini_cluster.cc:1428] Setting key 4b41343490c569b7a030fbebef11798f
I20260809 15:06:49.492307 32428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:49.510479 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.494545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.27.186.190:35125)
0809 15:06:49.494687 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:49.494692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:49.495929 (+ 1237us) server_negotiation.cc:407] Connection header received
0809 15:06:49.497636 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:49.497642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:49.497715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:49.497810 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:49.499167 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.499845 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.501231 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.501653 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.501773 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.504824 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:49.504828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:49.504832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:49.504864 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:49.507594 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.508371 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.508375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.508378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.508504 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.508908 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.508911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.508913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.509172 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:49.509294 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:49.510091 (+ 797us) server_negotiation.cc:299] Negotiation successful
0809 15:06:49.510258 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 15:06:49.511538 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.495108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:35125 (local address 127.0.0.1:58622)
0809 15:06:49.495795 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.495828 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:49.497387 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:49.498058 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:49.498068 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:49.498529 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:49.499005 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.499021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.499992 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.499996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.501094 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.501107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.501862 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.501869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.502117 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.502646 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:49.502676 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:49.504614 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:49.507721 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.507729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.507743 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.508256 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.508607 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.508612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.508616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.508821 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.509298 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:49.509305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:49.509695 (+ 390us) client_negotiation.cc:769] Sending connection context
0809 15:06:49.509988 (+ 293us) client_negotiation.cc:240] Negotiation successful
0809 15:06:49.510421 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":304,"threads_started":1}
I20260809 15:06:49.513036 32428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.186.188@KRBTEST.COM; defaulting to no policy
I20260809 15:06:49.526542 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.513560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40816)
0809 15:06:49.513704 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.513725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:49.513805 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:49.514858 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:49.514867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:49.515310 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:49.515472 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.515482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.516760 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.516765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.517944 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.517958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.518719 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.518725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.518866 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.519336 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:49.519354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:49.521585 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:49.524419 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.524424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.524428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.524885 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.525222 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.525226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.525234 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.525373 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.525872 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:49.525875 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:49.525977 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 15:06:49.526110 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 15:06:49.526306 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 15:06:49.526683 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.513937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.27.186.189:37051)
0809 15:06:49.514489 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:49.514495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:49.514519 (+ 24us) server_negotiation.cc:407] Connection header received
0809 15:06:49.514557 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:49.514562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:49.514702 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:49.514808 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:49.515635 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.516626 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.518094 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.518575 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.518662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.521777 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:49.521789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:49.521799 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:49.521826 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:49.524272 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.525005 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.525010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.525012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.525121 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.525460 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.525463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.525465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.525754 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:49.525912 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:49.526225 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 15:06:49.526395 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":308,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.188@KRBTEST.COM" created.
I20260809 15:06:49.528642 32428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:49.529543 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.529170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54920)
0809 15:06:49.529318 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.529419 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 15:06:49.529778 32357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.531260 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.530952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54928)
0809 15:06:49.531094 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.531164 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 15:06:49.531442 32428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab.
Entry for principal kudu/127.27.186.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab.
I20260809 15:06:49.554530 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:35627
--webserver_interface=127.27.186.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab
--principal=kudu/127.27.186.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:35125,127.27.186.189:37051,127.27.186.188:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:06:49.580703 32428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 1
I20260809 15:06:49.581678 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.581235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54932)
0809 15:06:49.581413 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.581518 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 15:06:49.582630 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.582264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54936)
0809 15:06:49.582423 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.582508 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 15:06:49.582829 32428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.651031 32428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 2
I20260809 15:06:49.651995 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.651557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54950)
0809 15:06:49.651734 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.651831 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 15:06:49.652897 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.652555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54964)
0809 15:06:49.652682 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.652763 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 15:06:49.653088 32428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.709111 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } attempt: 5
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:06:49.722903 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.709862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.27.186.189:37051)
0809 15:06:49.710046 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:49.710052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:49.710089 (+ 37us) server_negotiation.cc:407] Connection header received
0809 15:06:49.710313 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:49.710316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:49.710391 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:49.710473 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:49.711429 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.712143 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.713869 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.714336 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.714427 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.717968 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:49.717975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:49.717979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:49.718038 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:49.720438 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.721168 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.721171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.721174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.721294 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:49.721696 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:49.721701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:49.721704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:49.721978 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:49.722132 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:49.722535 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 15:06:49.722690 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 15:06:49.723124 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.709740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40830)
0809 15:06:49.709983 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.710006 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:49.710177 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:49.710544 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:49.710549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:49.711058 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:49.711292 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.711306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.712286 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.712292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.713737 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.713754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.714475 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.714482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.714692 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.715304 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:49.715326 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:49.717786 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:49.720577 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.720582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.720586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.721066 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.721392 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:49.721397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:49.721401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:49.721590 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:49.722115 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:49.722120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:49.722299 (+ 179us) client_negotiation.cc:769] Sending connection context
0809 15:06:49.722456 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 15:06:49.722767 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 15:06:49.724100 32321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:49.725057 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.724636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54980)
0809 15:06:49.724803 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.724898 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 15:06:49.725975 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.725629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54988)
0809 15:06:49.725757 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.725836 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 15:06:49.726227 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.744306 32428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 3
I20260809 15:06:49.745280 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.744787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:54998)
0809 15:06:49.744998 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.745095 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 15:06:49.746356 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.745862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55008)
0809 15:06:49.746105 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.746201 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260809 15:06:49.746573 32428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.777417 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 1
I20260809 15:06:49.778378 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.777880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55010)
0809 15:06:49.778096 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.778199 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 15:06:49.779305 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.778947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55026)
0809 15:06:49.779080 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.779159 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 15:06:49.779525 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.847728 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 2
I20260809 15:06:49.848883 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.848315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55036)
0809 15:06:49.848559 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.848683 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 15:06:49.849881 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.849563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55038)
0809 15:06:49.849684 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.849769 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 15:06:49.850153 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
W20260809 15:06:49.874285 32438 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 15:06:49.874579 32438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:49.874668 32438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:49.874780 32438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:49.881704 32438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:49.881791 32438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:49.881882 32438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:49.881945 32438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:49.882043 32438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:49.889757 32428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 4
I20260809 15:06:49.890827 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.890308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55046)
0809 15:06:49.890541 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.890639 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 15:06:49.891767 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.891407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55052)
0809 15:06:49.891548 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.891634 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 15:06:49.891750 32438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:35125,127.27.186.189:37051,127.27.186.188:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab
--principal=kudu/127.27.186.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.186.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
W20260809 15:06:49.892040 32428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.892889 32438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:49.894107 32438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:49.904515 32446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:49.904572 32450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:49.904594 32447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:49.905583 32438 server_base.cc:1034] running on GCE node
I20260809 15:06:49.906261 32438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:49.907404 32438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:49.908605 32438 hybrid_clock.cc:648] HybridClock initialized: now 1786288009908546 us; error 31 us; skew 500 ppm
I20260809 15:06:49.908898 32438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:49.911736 32438 init.cc:377] Logged in from keytab as kudu/127.27.186.188@KRBTEST.COM (short username kudu)
I20260809 15:06:49.912951 32438 webserver.cc:460] Webserver started at http://127.27.186.188:40885/ using document root <none> and password file <none>
I20260809 15:06:49.913549 32438 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:49.913633 32438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:49.913924 32438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:49.916605 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data/instance:
uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b"
format_stamp: "Formatted at 2026-08-09 15:06:49 on dist-test-slave-q9h9"
server_key: "2d542424267f6bf37967bfeb070c2a16"
server_key_iv: "a1832a93537f845be894f82305dbefc4"
server_key_version: "encryptionkey@0"
I20260809 15:06:49.917366 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal/instance:
uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b"
format_stamp: "Formatted at 2026-08-09 15:06:49 on dist-test-slave-q9h9"
server_key: "2d542424267f6bf37967bfeb070c2a16"
server_key_iv: "a1832a93537f845be894f82305dbefc4"
server_key_version: "encryptionkey@0"
I20260809 15:06:49.922086 32438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 15:06:49.925294 32462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:49.926470 32438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:06:49.926604 32438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal
uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b"
format_stamp: "Formatted at 2026-08-09 15:06:49 on dist-test-slave-q9h9"
server_key: "2d542424267f6bf37967bfeb070c2a16"
server_key_iv: "a1832a93537f845be894f82305dbefc4"
server_key_version: "encryptionkey@0"
I20260809 15:06:49.926767 32438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:49.954062 32438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:49.956341 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 3
I20260809 15:06:49.957321 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.956854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55066)
0809 15:06:49.957038 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.957142 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 15:06:49.958341 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.957939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55068)
0809 15:06:49.958125 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.958212 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 15:06:49.958341 32438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20260809 15:06:49.958555 32321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:35627: Network error: Client connection negotiation failed: client connection to 127.27.186.188:35627: connect: Connection refused (error 111)
I20260809 15:06:49.958585 32438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:49.974738 32438 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.188:35627
I20260809 15:06:49.974757 32554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.188:35627 every 8 connection(s)
I20260809 15:06:49.975898 32438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
I20260809 15:06:49.979153 32555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:49.984947 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 32438
I20260809 15:06:49.985100 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786287660051758-28394-0/minicluster-data/master-2/wal/instance
I20260809 15:06:49.985505 28394 external_mini_cluster.cc:1428] Setting key 077e0e0e0c5541d9534d95c12d26003c
I20260809 15:06:49.985011 32555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
Aug 09 15:06:49 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:06:50.003854 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.987635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.27.186.190:35125)
0809 15:06:49.987811 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:49.987817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:49.988593 (+ 776us) server_negotiation.cc:407] Connection header received
0809 15:06:49.989871 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:49.989875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:49.989935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:49.990010 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:49.991312 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.992054 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.993710 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.994387 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.994473 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.998131 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:49.998138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:49.998143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:49.998181 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.000787 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.001540 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.001546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.001550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.001738 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.002165 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.002171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.002174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.002510 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.002658 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.003368 (+ 710us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.003586 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:50.004791 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.987752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:35125 (local address 127.0.0.1:58636)
0809 15:06:49.988448 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:49.988502 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:49.989620 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:49.990122 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:49.990128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:49.990816 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:49.991142 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.991158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.992204 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.992209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.993579 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:49.993590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.994599 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.994606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:49.994932 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.995571 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:49.995591 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:49.997938 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:50.000890 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.000899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.000906 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.001425 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.001851 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.001854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.001856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.002064 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.002630 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:50.002636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:50.002991 (+ 355us) client_negotiation.cc:769] Sending connection context
0809 15:06:50.003262 (+ 271us) client_negotiation.cc:240] Negotiation successful
0809 15:06:50.003620 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":30,"thread_start_us":300,"threads_started":1}
I20260809 15:06:50.005393 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:49.987057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.27.186.188:35627)
0809 15:06:49.987722 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:49.987734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:49.987778 (+ 44us) server_negotiation.cc:407] Connection header received
0809 15:06:49.989611 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:49.989637 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:49.989913 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:49.990111 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:49.991480 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.992881 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.994300 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:49.995073 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:49.995203 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:49.999475 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:49.999482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:49.999492 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:49.999519 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.002473 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.003283 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.003291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.003294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.003460 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.003869 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.003874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.003877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.004241 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.004463 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.004787 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.004987 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":469,"thread_start_us":273,"threads_started":1}
I20260809 15:06:50.006299 32555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:50 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
Aug 09 15:06:50 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:50.019766 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.006863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:37051 (local address 127.0.0.1:40846)
0809 15:06:50.007014 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:50.007030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:50.007122 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:50.007420 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:50.007424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:50.007758 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:50.007928 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.007939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.008911 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.008916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.009744 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.009755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.010489 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.010496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.010676 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.011112 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:50.011128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:50.013438 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:50.017110 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.017115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.017119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.017583 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.018245 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.018250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.018254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.018428 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.019041 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:50.019046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:50.019204 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 15:06:50.019357 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 15:06:50.019553 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 15:06:50.020542 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.006942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.27.186.189:37051)
0809 15:06:50.007109 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.007114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.007132 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:06:50.007228 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.007233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.007299 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.007393 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.008125 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.008781 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.009878 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.010360 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.010468 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.013629 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.013635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.013640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.013677 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.016955 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.017914 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.017919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.017923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.018103 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.018552 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.018556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.018560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.018889 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.019026 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.020019 (+ 993us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.020195 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:06:50.021473 32555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:50.021842 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.007927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.27.186.190:35125)
0809 15:06:50.008063 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.008068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.008135 (+ 67us) server_negotiation.cc:407] Connection header received
0809 15:06:50.008299 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.008304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.008360 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.008431 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.009529 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.010260 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.011388 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.011863 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.011972 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.016166 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.016171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.016174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.016200 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.019228 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.019939 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.019944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.019947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.020097 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.020499 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.020504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.020507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.020823 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.021302 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.021419 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.021570 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 15:06:50 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
Aug 09 15:06:50 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:06:50.034569 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.022316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.27.186.188:35627)
0809 15:06:50.022461 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.022466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.022480 (+ 14us) server_negotiation.cc:407] Connection header received
0809 15:06:50.022513 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.022515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.022559 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.022642 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.023378 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.024302 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.025707 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.026270 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.026390 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.029264 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.029271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.029282 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.029316 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.032122 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.032781 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.032784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.032787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.032902 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.033288 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.033291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.033293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.033605 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.033706 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.034161 (+ 455us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.034351 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":54}
I20260809 15:06:50.034919 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.022078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55090)
0809 15:06:50.022303 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:50.022318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:50.022396 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:50.022675 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:50.022680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:50.023081 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:50.023247 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.023259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.024423 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.024426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.025563 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.025577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.026409 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.026415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.026610 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.027066 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:50.027080 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:50.029093 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:50.032242 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.032247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.032251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.032670 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.033009 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.033014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.033018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.033171 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.033719 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:50.033724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:50.033882 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 15:06:50.034088 (+ 206us) client_negotiation.cc:240] Negotiation successful
0809 15:06:50.034518 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":119}
I20260809 15:06:50.037379 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.023350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.27.186.189:37051)
0809 15:06:50.023484 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.023487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.023500 (+ 13us) server_negotiation.cc:407] Connection header received
0809 15:06:50.023534 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.023537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.023580 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.023645 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.024932 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.025876 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.027141 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.027538 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.027643 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.031992 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.031997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.031999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.032027 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.034990 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.035623 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.035627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.035629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.035768 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.036131 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.036136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.036140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.036386 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.036630 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.036894 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.037087 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 15:06:50.039652 32555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: Bootstrap starting.
I20260809 15:06:50.041630 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:50.042908 32555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:50.043905 32555 log.cc:824] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:50.046401 32555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: No bootstrap required, opened a new log
I20260809 15:06:50.044167 32278 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b6d871e9721e4638ad1233faeb5a3e1f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644
I20260809 15:06:50.052945 32555 raft_consensus.cc:348] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.053212 32555 raft_consensus.cc:729] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c926ce8cf771419bbb6ab5f4c14fdd4b, State: Initialized, Role: FOLLOWER
I20260809 15:06:50.053591 32555 consensus_queue.cc:260] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.054358 32563 sys_catalog.cc:455] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.054636 32563 sys_catalog.cc:458] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.055328 32555 sys_catalog.cc:564] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: configured and running, proceeding with master startup.
W20260809 15:06:50.063901 32579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:06:50.064011 32579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:06:50.061755 32438 master_runner.cc:186] Error getting master registration for 127.27.186.190:35125: OK, instance_id { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" instance_seqno: 1786288009007370 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 15:06:50.069483 32438 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 15:06:50.135749 32321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 4
Aug 09 15:06:50 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:06:50.151556 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.136438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55092)
0809 15:06:50.136657 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:50.136677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:50.136804 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:50.137132 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:50.137137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:50.137640 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:50.137852 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.137864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.139350 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.139356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.140845 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.140859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.141857 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.141862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.142091 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.142684 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:50.142702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:50.144844 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:50.148798 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.148804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.148808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.149285 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.149696 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.149700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.149704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.149890 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.150600 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:50.150606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:50.150762 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 15:06:50.150916 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 15:06:50.151180 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 15:06:50.151626 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.136614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.27.186.188:35627)
0809 15:06:50.136811 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.136818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.136843 (+ 25us) server_negotiation.cc:407] Connection header received
0809 15:06:50.136908 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.136914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.136992 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.137104 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.138049 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.139215 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.140993 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.141721 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.141858 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.145047 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.145053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.145058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.145097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.148659 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.149404 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.149409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.149413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.149587 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.149992 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.149997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.150000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.150477 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.150635 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.151021 (+ 386us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.151240 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:06:50.157760 32321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f: Bootstrap starting.
I20260809 15:06:50.160985 32321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:50.162001 32321 log.cc:824] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:50.164635 32321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f: No bootstrap required, opened a new log
I20260809 15:06:50.171809 32321 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.172200 32321 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d871e9721e4638ad1233faeb5a3e1f, State: Initialized, Role: FOLLOWER
I20260809 15:06:50.172617 32321 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.173489 32584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.173949 32584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.174643 32321 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:50.174908 32278 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:06:50.175107 32278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:50.178706 32278 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.180608 32278 leader_election.cc:288] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [CANDIDATE]: Term 1 election: Requested vote from peers aaedf61362c74b65a49a88825eff2000 (127.27.186.189:37051), c926ce8cf771419bbb6ab5f4c14fdd4b (127.27.186.188:35627)
I20260809 15:06:50.181377 32514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b"
I20260809 15:06:50.181792 32514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:50.184293 32514 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d871e9721e4638ad1233faeb5a3e1f in term 1.
I20260809 15:06:50.184914 32245 leader_election.cc:302] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6d871e9721e4638ad1233faeb5a3e1f, c926ce8cf771419bbb6ab5f4c14fdd4b; no voters:
I20260809 15:06:50.185351 32584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:50.189746 32584 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [term 1 LEADER]: Becoming Leader. State: Replica: b6d871e9721e4638ad1233faeb5a3e1f, State: Running, Role: LEADER
I20260809 15:06:50.190361 32584 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.191222 32428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } attempt: 5
Aug 09 15:06:50 dist-test-slave-q9h9 krb5kdc[32197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:06:50.203696 32610 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d871e9721e4638ad1233faeb5a3e1f. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.204089 32610 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:50.206570 32613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:50.209299 32615 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260809 15:06:50.210271 32470 tablet.cc:2369] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:50.212615 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.191782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:35627 (local address 127.0.0.1:55108)
0809 15:06:50.191977 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:50.191997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:50.192117 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:50.192594 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:50.192599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:50.193128 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:50.193366 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.193379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.195017 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.195024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.196490 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:50.196507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.197790 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.197797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:50.198038 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.198742 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:50.198791 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:50.202651 (+ 3860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:50.208391 (+ 5740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.208397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.208402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.208931 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.209657 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:50.209661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:50.209665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:50.209883 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:50.211631 (+ 1748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:50.211636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:50.211813 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 15:06:50.211940 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 15:06:50.212218 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 15:06:50.213797 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.192098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.27.186.188:35627)
0809 15:06:50.192270 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.192275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.192294 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:06:50.192350 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.192355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.192446 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.192550 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.193561 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.194799 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.196728 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.197424 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.197541 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.202858 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.202864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.202868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.202907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.208185 (+ 5278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.209165 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.209173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.209176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.209524 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.211130 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.211135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.211138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.211488 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.213084 (+ 1596us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.213206 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.213427 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 15:06:50.219550 32428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: Bootstrap starting.
I20260809 15:06:50.221937 32613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:50.223393 32428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:50.213255 32215 master_runner.cc:186] Error getting master registration for 127.27.186.189:37051: OK, instance_id { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" instance_seqno: 1786288009477590 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 15:06:50.224699 32428 log.cc:824] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:50.224879 32215 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 15:06:50.233960 32428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: No bootstrap required, opened a new log
I20260809 15:06:50.246943 32514 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [term 1 FOLLOWER]: Refusing update from remote peer b6d871e9721e4638ad1233faeb5a3e1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:06:50.247510 32248 tablet.cc:2369] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:50.247902 32610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:50.258543 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.211782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.27.186.190:35125)
0809 15:06:50.211941 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.211947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.223036 (+ 11089us) server_negotiation.cc:407] Connection header received
0809 15:06:50.223236 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.223241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.223313 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.223401 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.228984 (+ 5583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.230201 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.242772 (+ 12571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.243533 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.243672 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.249803 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.249810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.249815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.249855 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.253603 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.254997 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.255002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.255005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.255190 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.255928 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.255933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.255937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.256295 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.256558 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.257536 (+ 978us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.257777 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 15:06:50.262010 32428 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.262483 32428 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaedf61362c74b65a49a88825eff2000, State: Initialized, Role: FOLLOWER
I20260809 15:06:50.263168 32428 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } }
I20260809 15:06:50.263833 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.223797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.27.186.189:37051)
0809 15:06:50.223970 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.223975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.234574 (+ 10599us) server_negotiation.cc:407] Connection header received
0809 15:06:50.234848 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.234853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.234934 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.235012 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.237257 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.238442 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.242131 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.242705 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.242804 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.255380 (+ 12576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.255387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.255392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.255440 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.259305 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.260175 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.260180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.260194 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.260375 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.260888 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.260894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.260897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.261558 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.262053 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.262545 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.263054 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 15:06:50.275363 32563 sys_catalog.cc:455] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d871e9721e4638ad1233faeb5a3e1f. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.275683 32563 sys_catalog.cc:458] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.276759 32626 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.277169 32626 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.285184 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.234655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.27.186.188:35627)
0809 15:06:50.234865 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.234870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.252242 (+ 17372us) server_negotiation.cc:407] Connection header received
0809 15:06:50.254993 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.254998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.255093 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.255180 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.258682 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.259717 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.265900 (+ 6183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.266681 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.266809 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.277342 (+ 10533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:50.277348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:50.277353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:50.277397 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:50.281252 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.282130 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.282135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.282138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.282359 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:50.282836 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:50.282841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:50.282844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:50.283196 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:50.284352 (+ 1156us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.284523 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.284786 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 15:06:50.285753 32584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.286100 32584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:50.287760 32428 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:50.287847 32613 catalog_manager.cc:1347] Generated new cluster ID: 8a4c23f6ea134041a8bdab9ae95b24dc
I20260809 15:06:50.287966 32613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:50.294090 32389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:50.294159 32563 sys_catalog.cc:455] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.294401 32563 sys_catalog.cc:458] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.297729 32389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [term 1 FOLLOWER]: Refusing update from remote peer b6d871e9721e4638ad1233faeb5a3e1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:50.298388 32390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aaedf61362c74b65a49a88825eff2000"
I20260809 15:06:50.298693 32610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:50.301764 32390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d871e9721e4638ad1233faeb5a3e1f in term 1.
I20260809 15:06:50.320401 32613 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:50.322060 32613 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:50.332724 32613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f: Generated new TSK 0
I20260809 15:06:50.333675 32613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:06:50.370388 32626 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d871e9721e4638ad1233faeb5a3e1f. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.370699 32626 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.370522 32584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.370824 32584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d871e9721e4638ad1233faeb5a3e1f [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:50.371025 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d871e9721e4638ad1233faeb5a3e1f" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 35125 } } peers { permanent_uuid: "aaedf61362c74b65a49a88825eff2000" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 37051 } } peers { permanent_uuid: "c926ce8cf771419bbb6ab5f4c14fdd4b" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 35627 } } }
I20260809 15:06:50.371356 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:50.404688 32653 catalog_manager.cc:1259] Loaded cluster ID: 8a4c23f6ea134041a8bdab9ae95b24dc
I20260809 15:06:50.404870 32653 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: loading cluster ID for follower catalog manager: success
I20260809 15:06:50.408411 32653 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: acquiring CA information for follower catalog manager: success
I20260809 15:06:50.420435 32653 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aaedf61362c74b65a49a88825eff2000: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 15:06:50.449888 28394 mini_kdc.cc:348] Time spent kdestroy: real 0.107s user 0.004s sys 0.024s
W20260809 15:06:50.504324 32323 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.492598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.27.186.190:35125)
0809 15:06:50.492842 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.492849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.502576 (+ 9727us) server_negotiation.cc:407] Connection header received
0809 15:06:50.502866 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.502871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.502948 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.503022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.504099 (+ 1077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58672: BlockingRecv error: recv got EOF from 127.0.0.1:58672 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 15:06:50.506623 32434 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.502722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.27.186.189:37051)
0809 15:06:50.502962 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.502967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.505067 (+ 2100us) server_negotiation.cc:407] Connection header received
0809 15:06:50.505299 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.505305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.505398 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.505468 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.506430 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40864: BlockingRecv error: recv got EOF from 127.0.0.1:40864 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260809 15:06:50.512241 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.509858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.27.186.188:35627)
0809 15:06:50.510071 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.510077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.510564 (+ 487us) server_negotiation.cc:407] Connection header received
0809 15:06:50.510812 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.510817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.510939 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.511018 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.512005 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55128: BlockingRecv error: recv got EOF from 127.0.0.1:55128 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:06:50.565390 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.551849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.27.186.189:37051)
0809 15:06:50.552014 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.552020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.557916 (+ 5896us) server_negotiation.cc:407] Connection header received
0809 15:06:50.558184 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.558190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.558365 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.558516 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.559226 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.560411 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.562481 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.563139 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.563236 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.563970 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.564579 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.564723 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.564967 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.565060 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 15:06:50.578800 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.576751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.27.186.188:35627)
0809 15:06:50.576916 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.576922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.577378 (+ 456us) server_negotiation.cc:407] Connection header received
0809 15:06:50.577547 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.577552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.577662 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.577768 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.578580 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55134: BlockingRecv error: recv got EOF from 127.0.0.1:55134 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:06:50.596524 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.590137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.27.186.189:37051)
0809 15:06:50.590319 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.590324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.590783 (+ 459us) server_negotiation.cc:407] Connection header received
0809 15:06:50.590938 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.590943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.591033 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.591121 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.591508 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.592642 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.594561 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.595229 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.595364 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.595789 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.595945 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.596038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.596176 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.596272 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 15:06:50.609886 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.607982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.27.186.188:35627)
0809 15:06:50.608142 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.608147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.608607 (+ 460us) server_negotiation.cc:407] Connection header received
0809 15:06:50.608766 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.608771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.608861 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.608958 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.609686 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55136: BlockingRecv error: recv got EOF from 127.0.0.1:55136 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:50.621721 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.615293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.27.186.189:37051)
0809 15:06:50.615453 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.615458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.615969 (+ 511us) server_negotiation.cc:407] Connection header received
0809 15:06:50.616117 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.616122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.616194 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.616297 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.616686 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.617782 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.619696 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.620398 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.620537 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.620973 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.621118 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.621208 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.621354 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.621427 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 15:06:50.632982 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.631239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.27.186.188:35627)
0809 15:06:50.631407 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.631412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.631544 (+ 132us) server_negotiation.cc:407] Connection header received
0809 15:06:50.631801 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.631806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.631911 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.632000 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.632766 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55148: BlockingRecv error: recv got EOF from 127.0.0.1:55148 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 15:06:50.648499 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.642057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.27.186.189:37051)
0809 15:06:50.642274 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.642279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.642750 (+ 471us) server_negotiation.cc:407] Connection header received
0809 15:06:50.642933 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.642938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.643022 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.643112 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.643498 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.644617 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.646495 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.647233 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.647358 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.647824 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.647953 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.648040 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.648178 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.648253 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260809 15:06:50.665562 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.663275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.27.186.188:35627)
0809 15:06:50.663717 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.663722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.664203 (+ 481us) server_negotiation.cc:407] Connection header received
0809 15:06:50.664352 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.664357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.664448 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.664551 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.665357 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55154: BlockingRecv error: recv got EOF from 127.0.0.1:55154 (error 108)
Metrics: {"server-negotiator.queue_time_us":317}
I20260809 15:06:50.685945 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.679293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.27.186.189:37051)
0809 15:06:50.679468 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.679473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.680002 (+ 529us) server_negotiation.cc:407] Connection header received
0809 15:06:50.680156 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.680161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.680261 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.680363 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.680801 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.681931 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.683840 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.684529 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.684661 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.685129 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.685283 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.685380 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.685521 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.685604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 15:06:50.696063 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.691846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.27.186.188:35627)
0809 15:06:50.692001 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.692006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.694590 (+ 2584us) server_negotiation.cc:407] Connection header received
0809 15:06:50.694777 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.694783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.694892 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.695000 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.695847 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55164: BlockingRecv error: recv got EOF from 127.0.0.1:55164 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:50.721917 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.715161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.27.186.189:37051)
0809 15:06:50.715329 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.715335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.715864 (+ 529us) server_negotiation.cc:407] Connection header received
0809 15:06:50.716039 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.716045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.716138 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.716256 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.716730 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.717980 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.719885 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.720526 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.720667 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.721186 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.721340 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.721447 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.721590 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.721677 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 15:06:50.730917 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.728575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.27.186.188:35627)
0809 15:06:50.728755 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.728761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.729289 (+ 528us) server_negotiation.cc:407] Connection header received
0809 15:06:50.729519 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.729525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.729612 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.729722 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.730701 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55172: BlockingRecv error: recv got EOF from 127.0.0.1:55172 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 15:06:50.779145 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.771882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.27.186.189:37051)
0809 15:06:50.772083 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.772089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.772660 (+ 571us) server_negotiation.cc:407] Connection header received
0809 15:06:50.772851 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.772858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.772997 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.773121 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.773628 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.774859 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.776936 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.777623 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.777789 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.778260 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.778396 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.778486 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.778730 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.778814 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 15:06:50.787681 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.785380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.27.186.188:35627)
0809 15:06:50.785574 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.785580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.786118 (+ 538us) server_negotiation.cc:407] Connection header received
0809 15:06:50.786302 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.786308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.786412 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.786508 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.787423 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55188: BlockingRecv error: recv got EOF from 127.0.0.1:55188 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:06:50.860328 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.855047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.27.186.189:37051)
0809 15:06:50.855237 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.855243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.855726 (+ 483us) server_negotiation.cc:407] Connection header received
0809 15:06:50.855890 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.855893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.855961 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.856063 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.856485 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.857281 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.858692 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.859131 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:50.859252 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:50.859648 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:50.859777 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:50.859851 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:50.859999 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 15:06:50.860075 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 15:06:50.866895 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:50.865164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.27.186.188:35627)
0809 15:06:50.865337 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.865341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.865726 (+ 385us) server_negotiation.cc:407] Connection header received
0809 15:06:50.865853 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.865856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.865923 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.866001 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:50.866681 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55204: BlockingRecv error: recv got EOF from 127.0.0.1:55204 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 15:06:51.002751 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:50.997936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.27.186.189:37051)
0809 15:06:50.998173 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:50.998180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:50.998402 (+ 222us) server_negotiation.cc:407] Connection header received
0809 15:06:50.998591 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:50.998594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:50.998661 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:50.998740 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:50.999170 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:50.999843 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.001164 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.001564 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.001674 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:51.002083 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:51.002188 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:51.002274 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:51.002413 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 15:06:51.002494 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260809 15:06:51.009618 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:51.007865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.27.186.188:35627)
0809 15:06:51.008068 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:51.008074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:51.008325 (+ 251us) server_negotiation.cc:407] Connection header received
0809 15:06:51.008504 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:51.008509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:51.008581 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:51.008663 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:51.009388 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55220: BlockingRecv error: recv got EOF from 127.0.0.1:55220 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 15:06:51.065847 32579 catalog_manager.cc:1259] Loaded cluster ID: 8a4c23f6ea134041a8bdab9ae95b24dc
I20260809 15:06:51.065936 32579 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: loading cluster ID for follower catalog manager: success
I20260809 15:06:51.068472 32579 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: acquiring CA information for follower catalog manager: success
I20260809 15:06:51.070181 32579 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c926ce8cf771419bbb6ab5f4c14fdd4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 15:06:51.273402 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:51.268936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.27.186.189:37051)
0809 15:06:51.269107 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:51.269111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:51.269389 (+ 278us) server_negotiation.cc:407] Connection header received
0809 15:06:51.269573 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:51.269577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:51.269647 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:51.269727 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:51.270058 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.270793 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.272080 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.272468 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.272564 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:51.272887 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:51.272990 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:51.273054 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:51.273146 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 15:06:51.273206 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:06:51.283389 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:51.278537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.27.186.188:35627)
0809 15:06:51.278749 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:51.278755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:51.278947 (+ 192us) server_negotiation.cc:407] Connection header received
0809 15:06:51.279089 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:51.279092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:51.279257 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:51.279353 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:51.279698 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.280348 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.281561 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.281926 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.282009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:51.282436 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:51.282929 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:51.283035 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:51.283074 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 15:06:51.283141 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 15:06:51.295322 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:51.289443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.27.186.189:37051)
0809 15:06:51.289599 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:51.289605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:51.289943 (+ 338us) server_negotiation.cc:407] Connection header received
0809 15:06:51.290247 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:51.290252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:51.290317 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:51.290362 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:51.290841 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.291766 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.293548 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.293977 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.294133 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:51.294579 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:51.294712 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:51.294806 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:51.294966 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 15:06:51.295043 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 15:06:51.297479 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:51.288844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.27.186.190:35125)
0809 15:06:51.291148 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:51.291154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:51.291173 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:06:51.291290 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:51.291295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:51.291466 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:51.291629 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:51.292511 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.293525 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.295358 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.295896 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.295989 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:51.296533 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:51.296931 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:51.297048 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:51.297151 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 15:06:51.297216 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2184,"thread_start_us":2066,"threads_started":1}
I20260809 15:06:51.298837 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:51.290048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.27.186.188:35627)
0809 15:06:51.290196 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:51.290202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:51.292557 (+ 2355us) server_negotiation.cc:407] Connection header received
0809 15:06:51.292722 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:51.292727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:51.292801 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:51.292886 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:51.293456 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.294495 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.297107 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:51.297635 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:51.297757 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:51.298201 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:51.298290 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:51.298368 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:51.298502 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 15:06:51.298573 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 15:06:51.302589 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32215
I20260809 15:06:51.303474 32307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:51.408375 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 32215
W20260809 15:06:51.430092 32467 connection.cc:535] server connection from 127.0.0.1:55092 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 15:06:51.430130 32358 connection.cc:535] server connection from 127.0.0.1:40830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 15:06:51.430809 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32335
I20260809 15:06:51.431794 32421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:51.534274 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 32335
I20260809 15:06:51.558699 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 32438
I20260809 15:06:51.559715 32546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:51.663404 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 32438
2026-08-09T15:06:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4529 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 15:06:51 dist-test-slave-q9h9 krb5kdc[392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 15:06:51 dist-test-slave-q9h9 krb5kdc[392](info): set up 2 sockets
Aug 09 15:06:51 dist-test-slave-q9h9 krb5kdc[392](info): commencing operation
krb5kdc: starting...
W20260809 15:06:52.871807 28394 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/test-user.keytab.
Aug 09 15:06:53 dist-test-slave-q9h9 krb5kdc[392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T15:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T15:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.186.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
Entry for principal kudu/127.27.186.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab.
I20260809 15:06:53.116755 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44773
--webserver_interface=127.27.186.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:44773,127.27.186.189:40057,127.27.186.188:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 15:06:53.429900 414 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 15:06:53.430202 414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:53.430300 414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:53.430402 414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:53.437397 414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:53.437469 414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:53.437552 414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:53.437611 414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:53.437703 414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:53.446797 414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:44773,127.27.186.189:40057,127.27.186.188:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.190.keytab
--principal=kudu/127.27.186.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.190:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.186.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:53.447791 414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:53.448947 414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:53.459681 429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:53.459745 424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:53.459724 425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:53.460403 414 server_base.cc:1034] running on GCE node
I20260809 15:06:53.461093 414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:53.462325 414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:53.463533 414 hybrid_clock.cc:648] HybridClock initialized: now 1786288013463471 us; error 33 us; skew 500 ppm
I20260809 15:06:53.463822 414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:53 dist-test-slave-q9h9 krb5kdc[392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:53.471236 414 init.cc:377] Logged in from keytab as kudu/127.27.186.190@KRBTEST.COM (short username kudu)
I20260809 15:06:53.472559 414 webserver.cc:460] Webserver started at http://127.27.186.190:42663/ using document root <none> and password file <none>
I20260809 15:06:53.473186 414 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:53.473287 414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:53.473583 414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:53.476305 414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data/instance:
uuid: "1d3f906bb0a3433d90e3305c8e523f83"
format_stamp: "Formatted at 2026-08-09 15:06:53 on dist-test-slave-q9h9"
server_key: "3ddc239e81dccd8b05d52c918ba2aa6d"
server_key_iv: "0adf9d5af2afb9099f3ac9e811a7f754"
server_key_version: "encryptionkey@0"
I20260809 15:06:53.477046 414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal/instance:
uuid: "1d3f906bb0a3433d90e3305c8e523f83"
format_stamp: "Formatted at 2026-08-09 15:06:53 on dist-test-slave-q9h9"
server_key: "3ddc239e81dccd8b05d52c918ba2aa6d"
server_key_iv: "0adf9d5af2afb9099f3ac9e811a7f754"
server_key_version: "encryptionkey@0"
I20260809 15:06:53.481731 414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260809 15:06:53.484936 439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:53.486052 414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 15:06:53.486225 414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal
uuid: "1d3f906bb0a3433d90e3305c8e523f83"
format_stamp: "Formatted at 2026-08-09 15:06:53 on dist-test-slave-q9h9"
server_key: "3ddc239e81dccd8b05d52c918ba2aa6d"
server_key_iv: "0adf9d5af2afb9099f3ac9e811a7f754"
server_key_version: "encryptionkey@0"
I20260809 15:06:53.486413 414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:53.506816 414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:53.510237 414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:53.510460 414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:53.527168 414 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.190:44773
I20260809 15:06:53.527187 527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.190:44773 every 8 connection(s)
I20260809 15:06:53.528306 414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/data/info.pb
I20260809 15:06:53.531551 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:53.537503 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:53.538358 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 414
I20260809 15:06:53.538535 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-0/wal/instance
I20260809 15:06:53.538949 28394 external_mini_cluster.cc:1428] Setting key 17f609b4abf6e7a12fff06bba1888047
Aug 09 15:06:53 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:53.557993 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.540320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44773 (local address 127.0.0.1:59188)
0809 15:06:53.541140 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.541194 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:53.542975 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:53.543809 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:53.543814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:53.544308 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:53.544624 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:53.544640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:53.546571 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:53.546575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:53.547603 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:53.547615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:53.548660 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:53.548664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:53.548789 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:53.549316 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:53.549336 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:53.551287 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:53.554344 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:53.554355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:53.554365 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:53.554732 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:53.555108 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:53.555113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:53.555117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:53.555260 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:53.555914 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:53.555921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:53.556121 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 15:06:53.556329 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 15:06:53.556874 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":163,"thread_start_us":349,"threads_started":1}
I20260809 15:06:53.558051 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.540319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.27.186.190:44773)
0809 15:06:53.541839 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:53.541850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:53.541899 (+ 49us) server_negotiation.cc:407] Connection header received
0809 15:06:53.543322 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:53.543328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:53.543597 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:53.543790 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:53.544822 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:53.546434 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:53.547737 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:53.548470 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:53.548633 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:53.551501 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:53.551511 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:53.551522 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:53.551560 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:53.554233 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:53.554858 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:53.554864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:53.554866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:53.555005 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:53.555359 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:53.555364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:53.555366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:53.555775 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:53.555956 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:53.556470 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 15:06:53.556747 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1326,"spinlock_wait_cycles":35200,"thread_start_us":1145,"threads_started":1}
I20260809 15:06:53.560688 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:53.561619 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.561264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45790)
0809 15:06:53.561412 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.561510 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 15:06:53.561885 447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
I20260809 15:06:53.563293 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.562984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45804)
0809 15:06:53.563108 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.563174 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:06:53.563493 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:40057: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.186.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab.
Entry for principal kudu/127.27.186.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab.
I20260809 15:06:53.598677 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:40057
--webserver_interface=127.27.186.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab
--principal=kudu/127.27.186.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:44773,127.27.186.189:40057,127.27.186.188:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:06:53.612763 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } attempt: 1
I20260809 15:06:53.613750 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.613317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45812)
0809 15:06:53.613485 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.613618 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 15:06:53.614765 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.614432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45822)
0809 15:06:53.614561 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.614629 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 15:06:53.614959 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:40057: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
I20260809 15:06:53.683209 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } attempt: 2
I20260809 15:06:53.684320 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.683877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45834)
0809 15:06:53.684091 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.684180 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 15:06:53.685319 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.684942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45838)
0809 15:06:53.685114 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.685184 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 15:06:53.685508 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:40057: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
I20260809 15:06:53.776736 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } attempt: 3
I20260809 15:06:53.777758 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.777344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45840)
0809 15:06:53.777529 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.777617 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 15:06:53.778785 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.778434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45842)
0809 15:06:53.778573 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.778640 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 15:06:53.778959 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:40057: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
W20260809 15:06:53.917909 543 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 15:06:53.918219 543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:53.918319 543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:53.918428 543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 15:06:53.922173 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } attempt: 4
I20260809 15:06:53.923197 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.922720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45848)
0809 15:06:53.922981 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.923063 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260809 15:06:53.924151 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:53.923812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45856)
0809 15:06:53.923955 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:53.924034 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:06:53.924338 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.189:40057: Network error: Client connection negotiation failed: client connection to 127.27.186.189:40057: connect: Connection refused (error 111)
W20260809 15:06:53.925586 543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:53.925671 543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:53.925760 543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:53.925832 543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:53.925918 543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:53.935123 543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:44773,127.27.186.189:40057,127.27.186.188:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.189.keytab
--principal=kudu/127.27.186.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.189:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.186.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:53.936169 543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:53.937376 543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:53.947949 553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:53.948035 552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:53.948027 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:53.948871 543 server_base.cc:1034] running on GCE node
I20260809 15:06:53.949538 543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:53.950745 543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:53.951968 543 hybrid_clock.cc:648] HybridClock initialized: now 1786288013951899 us; error 41 us; skew 500 ppm
I20260809 15:06:53.952260 543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:53 dist-test-slave-q9h9 krb5kdc[392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:53.955127 543 init.cc:377] Logged in from keytab as kudu/127.27.186.189@KRBTEST.COM (short username kudu)
I20260809 15:06:53.956523 543 webserver.cc:460] Webserver started at http://127.27.186.189:43873/ using document root <none> and password file <none>
I20260809 15:06:53.957141 543 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:53.957242 543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:53.957530 543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:53.960320 543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data/instance:
uuid: "b92c0cefd6ea44bc8845ca19228a9323"
format_stamp: "Formatted at 2026-08-09 15:06:53 on dist-test-slave-q9h9"
server_key: "5684d6c07b1db23a8cd528803da6ec7f"
server_key_iv: "a569e994008adff3353cbd6c6b3e37fc"
server_key_version: "encryptionkey@0"
I20260809 15:06:53.961073 543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal/instance:
uuid: "b92c0cefd6ea44bc8845ca19228a9323"
format_stamp: "Formatted at 2026-08-09 15:06:53 on dist-test-slave-q9h9"
server_key: "5684d6c07b1db23a8cd528803da6ec7f"
server_key_iv: "a569e994008adff3353cbd6c6b3e37fc"
server_key_version: "encryptionkey@0"
I20260809 15:06:53.965692 543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 15:06:53.968930 562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:53.970163 543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 15:06:53.970321 543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal
uuid: "b92c0cefd6ea44bc8845ca19228a9323"
format_stamp: "Formatted at 2026-08-09 15:06:53 on dist-test-slave-q9h9"
server_key: "5684d6c07b1db23a8cd528803da6ec7f"
server_key_iv: "a569e994008adff3353cbd6c6b3e37fc"
server_key_version: "encryptionkey@0"
I20260809 15:06:53.970510 543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:53.982129 543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:53.985507 543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:53.985734 543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:54.002691 543 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.189:40057
I20260809 15:06:54.002724 638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.189:40057 every 8 connection(s)
I20260809 15:06:54.003790 543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/data/info.pb
I20260809 15:06:54.006982 641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:54.010463 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 543
I20260809 15:06:54.010653 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-1/wal/instance
I20260809 15:06:54.011205 28394 external_mini_cluster.cc:1428] Setting key 7caefcea51379810a6ff02aa178cc655
I20260809 15:06:54.013325 641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:54.031934 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.015622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.27.186.190:44773)
0809 15:06:54.015774 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.015778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.016842 (+ 1064us) server_negotiation.cc:407] Connection header received
0809 15:06:54.018639 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.018645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.018729 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.018816 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.020167 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.020846 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.022303 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.022706 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.022832 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.026099 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.026105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.026109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.026147 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.029186 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.029831 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.029836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.029839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.029992 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.030379 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.030384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.030388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.030709 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.030857 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.031437 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.031642 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 15:06:54.032485 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.016027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44773 (local address 127.0.0.1:59202)
0809 15:06:54.016698 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.016733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.018392 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.019039 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.019046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.019515 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.019991 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.020007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.020996 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.021000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.022161 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.022174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.022925 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.022933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.023135 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.023690 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.023717 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.025882 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.029308 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.029315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.029324 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.029713 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.030123 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.030126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.030129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.030279 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.030819 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.030823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.031080 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.031280 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.031584 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260809 15:06:54.033982 641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.186.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.186.188@KRBTEST.COM" created.
I20260809 15:06:54.048135 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.034641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45858)
0809 15:06:54.034786 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.034803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.034904 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.035671 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.035675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.036073 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.036248 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.036261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.037612 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.037617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.038873 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.038889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.039648 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.039654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.039843 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.040344 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.040365 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.042558 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.045658 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.045662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.045666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.046170 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.046531 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.046536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.046539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.046688 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.047206 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.047210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.047350 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.047493 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.047737 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 15:06:54.048170 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.034750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.27.186.189:40057)
0809 15:06:54.035306 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.035312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.035335 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:06:54.035373 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.035378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.035518 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.035620 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.036415 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.037482 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.039000 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.039496 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.039592 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.042780 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.042793 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.042808 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.042848 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.045531 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.046280 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.046287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.046289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.046416 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.046788 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.046791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.046793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.047087 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.047238 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.047614 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.047840 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":418,"thread_start_us":272,"threads_started":1}
I20260809 15:06:54.050434 641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:54.051338 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.050976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45758)
0809 15:06:54.051105 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.051204 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:06:54.051569 565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.052897 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.052622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45764)
0809 15:06:54.052746 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.052811 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 15:06:54.053079 641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.186.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab.
Entry for principal kudu/127.27.186.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab.
I20260809 15:06:54.074760 28394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
/tmp/dist-test-taskvIvrqI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:39465
--webserver_interface=127.27.186.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab
--principal=kudu/127.27.186.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.186.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.186.190:44773,127.27.186.189:40057,127.27.186.188:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 15:06:54.102344 641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 1
I20260809 15:06:54.103392 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.102979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45774)
0809 15:06:54.103141 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.103253 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 15:06:54.104259 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.103933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45784)
0809 15:06:54.104083 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.104152 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 15:06:54.104442 641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.172646 641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 2
I20260809 15:06:54.173756 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.173274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45798)
0809 15:06:54.173464 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.173565 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 15:06:54.174758 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.174426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45806)
0809 15:06:54.174569 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.174641 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 15:06:54.174978 641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.223595 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } attempt: 5
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:06:54.237553 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.224502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.27.186.189:40057)
0809 15:06:54.224688 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.224694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.224718 (+ 24us) server_negotiation.cc:407] Connection header received
0809 15:06:54.224834 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.224840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.224911 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.225009 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.225893 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.226701 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.227929 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.228428 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.228551 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.232042 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.232047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.232051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.232097 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.234862 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.235524 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.235529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.235532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.235681 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.236070 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.236074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.236084 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.236379 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.236524 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.236964 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.237197 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:54.237637 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.224319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45872)
0809 15:06:54.224515 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.224537 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.224692 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.225075 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.225081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.225504 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.225716 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.225729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.226856 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.226862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.227767 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.227781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.228609 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.228616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.228853 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.229544 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.229564 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.231843 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.235005 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.235011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.235016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.235393 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.235798 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.235804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.235808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.235956 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.236503 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.236508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.236676 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.236822 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.237097 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 15:06:54.238760 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } has no permanent_uuid. Determining permanent_uuid...
I20260809 15:06:54.239764 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.239386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45810)
0809 15:06:54.239544 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.239631 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 15:06:54.240731 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.240409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45814)
0809 15:06:54.240534 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.240625 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 15:06:54.240952 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.266180 641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 3
I20260809 15:06:54.267289 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.266811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45828)
0809 15:06:54.267030 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.267124 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 15:06:54.268376 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.267966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45838)
0809 15:06:54.268190 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.268268 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260809 15:06:54.268581 641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.292142 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 1
I20260809 15:06:54.293164 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.292749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45850)
0809 15:06:54.292914 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.293002 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 15:06:54.294100 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.293746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45860)
0809 15:06:54.293890 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.293961 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 15:06:54.294291 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.362490 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 2
I20260809 15:06:54.363649 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.363174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45866)
0809 15:06:54.363391 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.363478 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 15:06:54.364616 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.364281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45874)
0809 15:06:54.364429 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.364510 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 15:06:54.364807 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
W20260809 15:06:54.397837 653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 15:06:54.398169 653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 15:06:54.398274 653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 15:06:54.398391 653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 15:06:54.405562 653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 15:06:54.405658 653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 15:06:54.405741 653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 15:06:54.405805 653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 15:06:54.405891 653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 15:06:54.411774 641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 4
I20260809 15:06:54.412891 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.412426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45878)
0809 15:06:54.412641 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.412739 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 15:06:54.413812 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.413484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45892)
0809 15:06:54.413626 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.413704 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 15:06:54.414034 641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.415776 653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.186.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.186.190:44773,127.27.186.189:40057,127.27.186.188:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/krb5kdc/kudu_127.27.186.188.keytab
--principal=kudu/127.27.186.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.186.188:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.186.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 14:07:21 UTC on ccycloud.quasar-pgucrk.root.comops.site
build id 4576546
ASAN enabled
I20260809 15:06:54.416965 653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:06:54.418252 653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:06:54.428982 664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:54.428943 660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:06:54.429020 659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:06:54.429797 653 server_base.cc:1034] running on GCE node
I20260809 15:06:54.430526 653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 15:06:54.431747 653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 15:06:54.432950 653 hybrid_clock.cc:648] HybridClock initialized: now 1786288014432890 us; error 33 us; skew 500 ppm
I20260809 15:06:54.433262 653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 15:06:54.440481 653 init.cc:377] Logged in from keytab as kudu/127.27.186.188@KRBTEST.COM (short username kudu)
I20260809 15:06:54.441821 653 webserver.cc:460] Webserver started at http://127.27.186.188:38615/ using document root <none> and password file <none>
I20260809 15:06:54.442471 653 fs_manager.cc:359] Metadata directory not provided
I20260809 15:06:54.442559 653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:06:54.442811 653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:06:54.445493 653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data/instance:
uuid: "b3cbb84d560f446fbe1b53737884de87"
format_stamp: "Formatted at 2026-08-09 15:06:54 on dist-test-slave-q9h9"
server_key: "8f7f45ce7f761f6f6695bbcbe00e7f04"
server_key_iv: "38095c259f878e0bb564a4bf423abdd4"
server_key_version: "encryptionkey@0"
I20260809 15:06:54.446236 653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal/instance:
uuid: "b3cbb84d560f446fbe1b53737884de87"
format_stamp: "Formatted at 2026-08-09 15:06:54 on dist-test-slave-q9h9"
server_key: "8f7f45ce7f761f6f6695bbcbe00e7f04"
server_key_iv: "38095c259f878e0bb564a4bf423abdd4"
server_key_version: "encryptionkey@0"
I20260809 15:06:54.450910 653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 15:06:54.453990 673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:06:54.455163 653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:06:54.455315 653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data,/tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal
uuid: "b3cbb84d560f446fbe1b53737884de87"
format_stamp: "Formatted at 2026-08-09 15:06:54 on dist-test-slave-q9h9"
server_key: "8f7f45ce7f761f6f6695bbcbe00e7f04"
server_key_iv: "38095c259f878e0bb564a4bf423abdd4"
server_key_version: "encryptionkey@0"
I20260809 15:06:54.455498 653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:06:54.471001 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 3
I20260809 15:06:54.471954 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.471524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45894)
0809 15:06:54.471725 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.471806 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 15:06:54.472908 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.472567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45908)
0809 15:06:54.472734 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.472809 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260809 15:06:54.473133 528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.186.188:39465: Network error: Client connection negotiation failed: client connection to 127.27.186.188:39465: connect: Connection refused (error 111)
I20260809 15:06:54.491668 653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:06:54.495532 653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:06:54.495767 653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:06:54.512921 653 rpc_server.cc:307] RPC server started. Bound to: 127.27.186.188:39465
I20260809 15:06:54.512941 757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.186.188:39465 every 8 connection(s)
I20260809 15:06:54.514165 653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/data/info.pb
I20260809 15:06:54.517475 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:06:54.518481 28394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu as pid 653
I20260809 15:06:54.518637 28394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvIvrqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786287660051758-28394-0/minicluster-data/master-2/wal/instance
I20260809 15:06:54.519058 28394 external_mini_cluster.cc:1428] Setting key a5556fe4555c35454cbf91e1ca24552e
I20260809 15:06:54.525360 758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
I20260809 15:06:54.544382 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.520696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.27.186.188:39465)
0809 15:06:54.521390 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.521401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.521439 (+ 38us) server_negotiation.cc:407] Connection header received
0809 15:06:54.523144 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.523163 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.523531 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.523872 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.525248 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.526554 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.528135 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.529199 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.529388 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.534707 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.534718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.534745 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.534790 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.539087 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.539841 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.539849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.539857 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.540020 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.540468 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.540473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.540476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.541101 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.541391 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.542741 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.543138 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":284,"threads_started":1}
I20260809 15:06:54.547443 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.527709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.190:44773 (local address 127.0.0.1:59216)
0809 15:06:54.529719 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.529762 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.529888 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.530569 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.530577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.531333 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.531645 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.531660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.533053 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.533058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.534577 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.534592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.535861 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.535872 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.536085 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.536735 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.536756 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.539520 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.544223 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.544235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.544257 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.544927 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.545419 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.545424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.545427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.545580 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.546330 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.546337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.546519 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.546720 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.547004 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1787,"thread_start_us":1663,"threads_started":1}
I20260809 15:06:54.547487 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.529164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.27.186.190:44773)
0809 15:06:54.530152 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.530158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.530176 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:06:54.530226 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.530231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.530298 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.530383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.531821 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.532847 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.534750 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.535446 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.535538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.539701 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.539705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.539708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.539738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.544096 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.545075 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.545081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.545085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.545266 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.545711 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.545716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.545720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.546068 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.546213 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.546845 (+ 632us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.547072 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828}
I20260809 15:06:54.549872 758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.190@KRBTEST.COM
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:06:54.571048 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.551144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.27.186.190:44773)
0809 15:06:54.551298 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.551304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.551441 (+ 137us) server_negotiation.cc:407] Connection header received
0809 15:06:54.551624 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.551629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.551699 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.551789 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.553335 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.554548 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.556212 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.556777 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.556871 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.563478 (+ 6607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.563485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.563490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.563529 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.567868 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.568626 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.568632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.568635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.568794 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.569262 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.569268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.569271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.569621 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.569852 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.570504 (+ 652us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.570732 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:54.575919 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.552499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.27.186.189:40057)
0809 15:06:54.552670 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.552676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.552759 (+ 83us) server_negotiation.cc:407] Connection header received
0809 15:06:54.552911 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.552917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.552992 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.553083 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.554068 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.555147 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.556878 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.557558 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.557652 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.568021 (+ 10369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.568028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.568033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.568076 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.571555 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.572757 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.572764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.572768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.572939 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.573479 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.573484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.573488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.573919 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.574593 (+ 674us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.575286 (+ 693us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.575518 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 15:06:54.576476 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.552425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.189:40057 (local address 127.0.0.1:45882)
0809 15:06:54.552678 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.552694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.552808 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.553257 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.553263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.553697 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.553871 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.553881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.555341 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.555346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.556708 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.556723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.557897 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.557903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.558108 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.565284 (+ 7176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.565308 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.567781 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.572091 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.572098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.572102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.572607 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.573150 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.573155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.573159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.573346 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.574697 (+ 1351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.574702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.574985 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.575858 (+ 873us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.576148 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260809 15:06:54.577673 758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.188@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.186.189@KRBTEST.COM
I20260809 15:06:54.596639 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.578586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.27.186.188:39465)
0809 15:06:54.578786 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.578792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.578810 (+ 18us) server_negotiation.cc:407] Connection header received
0809 15:06:54.578854 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.578859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.578923 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.579012 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.579900 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.580894 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.582544 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.583141 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.583244 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.589344 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.589353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.589358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.589392 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.593757 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.594571 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.594576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.594580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.594738 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.595134 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.595139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.595142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.595484 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.595613 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.596079 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.596305 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 15:06:54.597704 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.578411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45930)
0809 15:06:54.578564 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.578582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.578686 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.579158 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.579163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.579593 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.579761 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.579772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.581043 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.581048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.582400 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.582416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.583445 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.583450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.583625 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.586518 (+ 2893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.586541 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.589078 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.593897 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.593902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.593906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.594460 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.594875 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.594879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.594883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.595029 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.595674 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.595678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.595807 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.597049 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.597356 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 15:06:54.598450 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.573594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.27.186.189:40057)
0809 15:06:54.575013 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.575018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.575041 (+ 23us) server_negotiation.cc:407] Connection header received
0809 15:06:54.575211 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.575216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.575284 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.575370 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.577482 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.578650 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.580332 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.580944 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.581058 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.590981 (+ 9923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.590988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.590994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.591033 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.594639 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.595465 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.595471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.595474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.595617 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.596062 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.596067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.596071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.596433 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.596707 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.597796 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.598038 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1279,"thread_start_us":1077,"threads_started":1}
I20260809 15:06:54.602483 28394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 15:06:54.604132 758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: Bootstrap starting.
I20260809 15:06:54.604635 487 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d3f906bb0a3433d90e3305c8e523f83"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59232
I20260809 15:06:54.608155 758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:54.609524 758 log.cc:824] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:54.612787 758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: No bootstrap required, opened a new log
I20260809 15:06:54.623490 758 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.623925 758 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cbb84d560f446fbe1b53737884de87, State: Initialized, Role: FOLLOWER
I20260809 15:06:54.624441 758 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.625407 767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.625783 767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.626585 758 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 15:06:54.635903 782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 15:06:54.636040 782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 15:06:54.633821 653 master_runner.cc:186] Error getting master registration for 127.27.186.190:44773: OK, instance_id { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" instance_seqno: 1786288013522465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 15:06:54.641788 653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 15:06:54.650338 528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 4
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.190@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:06:54.664953 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.650866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45936)
0809 15:06:54.651108 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.651137 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.651245 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.651575 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.651581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.652074 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.652288 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.652300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.653296 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.653302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.654817 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.654843 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.655560 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.655566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.655764 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.656355 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.656399 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.658973 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.662459 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.662465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.662484 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.662959 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.663328 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.663333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.663352 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.663526 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.664039 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.664044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.664197 (+ 153us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.664340 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.664590 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260809 15:06:54.665455 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.651072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.27.186.188:39465)
0809 15:06:54.651223 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.651228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.651247 (+ 19us) server_negotiation.cc:407] Connection header received
0809 15:06:54.651384 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.651389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.651449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.651527 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.652428 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.653161 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.654965 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.655425 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.655532 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.659132 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.659139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.659143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.659175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.662335 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.663083 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.663089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.663093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.663229 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.663635 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.663640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.663644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.663922 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.664065 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.664565 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.664916 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 15:06:54.670819 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83: Bootstrap starting.
I20260809 15:06:54.674502 528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:54.675727 528 log.cc:824] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:54.678701 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83: No bootstrap required, opened a new log
I20260809 15:06:54.685710 528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.686066 528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3f906bb0a3433d90e3305c8e523f83, State: Initialized, Role: FOLLOWER
I20260809 15:06:54.686525 528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.687638 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.688243 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.688674 528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:54.688716 487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:06:54.688877 487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:54.692175 487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.695524 720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d3f906bb0a3433d90e3305c8e523f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b3cbb84d560f446fbe1b53737884de87"
I20260809 15:06:54.695997 720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:54.699559 720 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3f906bb0a3433d90e3305c8e523f83 in term 1.
I20260809 15:06:54.700265 442 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3f906bb0a3433d90e3305c8e523f83, b3cbb84d560f446fbe1b53737884de87; no voters:
I20260809 15:06:54.695006 487 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [CANDIDATE]: Term 1 election: Requested vote from peers b92c0cefd6ea44bc8845ca19228a9323 (127.27.186.189:40057), b3cbb84d560f446fbe1b53737884de87 (127.27.186.188:39465)
I20260809 15:06:54.708678 788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:06:54.717454 641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } attempt: 5
I20260809 15:06:54.722529 788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3f906bb0a3433d90e3305c8e523f83, State: Running, Role: LEADER
I20260809 15:06:54.724080 788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
Aug 09 15:06:54 dist-test-slave-q9h9 krb5kdc[392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786288013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.186.189@KRBTEST.COM for kudu/127.27.186.188@KRBTEST.COM
I20260809 15:06:54.734367 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.718063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.186.188:39465 (local address 127.0.0.1:45946)
0809 15:06:54.718324 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 15:06:54.718343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 15:06:54.718463 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 15:06:54.718987 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 15:06:54.718993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 15:06:54.719531 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0809 15:06:54.719740 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.719754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.721240 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.721247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.722731 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 15:06:54.722747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.723893 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.723899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 15:06:54.724139 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.724820 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 15:06:54.724845 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 15:06:54.727721 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 15:06:54.731554 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.731560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.731564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.732043 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.732488 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 15:06:54.732493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 15:06:54.732497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 15:06:54.732655 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 15:06:54.733331 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 15:06:54.733337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 15:06:54.733533 (+ 196us) client_negotiation.cc:769] Sending connection context
0809 15:06:54.733670 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 15:06:54.733939 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260809 15:06:54.735244 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.718111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.27.186.188:39465)
0809 15:06:54.718301 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.718307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.718570 (+ 263us) server_negotiation.cc:407] Connection header received
0809 15:06:54.718631 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.718636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.718713 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.718799 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.719921 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.721051 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.722917 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.723585 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.723700 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.727932 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.727939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.727944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.727982 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.731383 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.732188 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.732194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.732197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.732358 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.732778 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.732784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.732787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.733108 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.733267 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.734671 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.734901 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 15:06:54.741689 641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: Bootstrap starting.
W20260809 15:06:54.747632 681 tablet.cc:2369] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:54.749455 641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: Neither blocks nor log segments found. Creating new log.
I20260809 15:06:54.751570 641 log.cc:824] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: Log is configured to *not* fsync() on all Append() calls
I20260809 15:06:54.759194 795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3f906bb0a3433d90e3305c8e523f83. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.759932 795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: This master's current role is: LEADER
W20260809 15:06:54.762874 453 tablet.cc:2369] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:06:54.769367 811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:06:54.772951 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.747994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.27.186.188:39465)
0809 15:06:54.748156 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.748161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.752069 (+ 3908us) server_negotiation.cc:407] Connection header received
0809 15:06:54.752343 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.752348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.752416 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.752480 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.754638 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.755713 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.759565 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.760249 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.760350 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.763966 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.763973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.763978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.764013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.767514 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.768332 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.768337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.768340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.768488 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.771200 (+ 2712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.771205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.771209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.771521 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.771778 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.772298 (+ 520us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.772536 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 15:06:54.777576 641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: No bootstrap required, opened a new log
I20260809 15:06:54.794380 811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:06:54.798683 641 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.799083 641 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92c0cefd6ea44bc8845ca19228a9323, State: Initialized, Role: FOLLOWER
I20260809 15:06:54.799592 641 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } }
I20260809 15:06:54.801584 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.733537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.27.186.190:44773)
0809 15:06:54.733738 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.733745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.747063 (+ 13318us) server_negotiation.cc:407] Connection header received
0809 15:06:54.750512 (+ 3449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.750517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.750591 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.750670 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.755103 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.756158 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.761433 (+ 5275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.762169 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.762265 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.772415 (+ 10150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.772421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.772426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.772463 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.780775 (+ 8312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.795439 (+ 14664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.795444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.795448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.795649 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.797437 (+ 1788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.797443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.797446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.797847 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.798170 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.800967 (+ 2797us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.801240 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 15:06:54.803097 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:54.743229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.27.186.189:40057)
0809 15:06:54.743403 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:54.743408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:54.747526 (+ 4118us) server_negotiation.cc:407] Connection header received
0809 15:06:54.750133 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:54.750139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:54.750218 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:54.750294 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:54.756333 (+ 6039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.757364 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.763284 (+ 5920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:54.763978 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:54.764075 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:54.770192 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 15:06:54.770199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 15:06:54.770205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 15:06:54.770244 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 15:06:54.779132 (+ 8888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.800744 (+ 21612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.800749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.800752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.800902 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 15:06:54.801366 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 15:06:54.801372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 15:06:54.801375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 15:06:54.801737 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 15:06:54.801960 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:54.802564 (+ 604us) server_negotiation.cc:299] Negotiation successful
0809 15:06:54.802801 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 15:06:54.822759 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.823164 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.824894 641 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:06:54.824921 603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d3f906bb0a3433d90e3305c8e523f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b92c0cefd6ea44bc8845ca19228a9323"
I20260809 15:06:54.825353 603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:06:54.828737 603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3f906bb0a3433d90e3305c8e523f83 in term 1.
I20260809 15:06:54.837101 720 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [term 1 FOLLOWER]: Refusing update from remote peer 1d3f906bb0a3433d90e3305c8e523f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:54.838160 795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:54.829870 414 master_runner.cc:186] Error getting master registration for 127.27.186.189:40057: OK, instance_id { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" instance_seqno: 1786288013998070 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 15:06:54.851361 414 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 15:06:54.854208 603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [term 1 FOLLOWER]: Refusing update from remote peer 1d3f906bb0a3433d90e3305c8e523f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:06:54.855023 795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:06:54.874537 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3f906bb0a3433d90e3305c8e523f83. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.874863 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.893122 824 mvcc.cc:205] Tried to move back new op lower bound from 7316635708763045888 to 7316635708349718528. Current Snapshot: MvccSnapshot[applied={T|T < 7316635708763045888}]
I20260809 15:06:54.898751 830 mvcc.cc:205] Tried to move back new op lower bound from 7316635708763045888 to 7316635708349718528. Current Snapshot: MvccSnapshot[applied={T|T < 7316635708763045888}]
I20260809 15:06:54.908830 767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3f906bb0a3433d90e3305c8e523f83. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.909165 767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.910964 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.911213 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.920852 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.921190 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:54.921579 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.921859 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83 [sys.catalog]: This master's current role is: LEADER
I20260809 15:06:54.923334 767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3f906bb0a3433d90e3305c8e523f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3f906bb0a3433d90e3305c8e523f83" member_type: VOTER last_known_addr { host: "127.27.186.190" port: 44773 } } peers { permanent_uuid: "b92c0cefd6ea44bc8845ca19228a9323" member_type: VOTER last_known_addr { host: "127.27.186.189" port: 40057 } } peers { permanent_uuid: "b3cbb84d560f446fbe1b53737884de87" member_type: VOTER last_known_addr { host: "127.27.186.188" port: 39465 } } }
I20260809 15:06:54.923575 767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 15:06:54.924402 829 mvcc.cc:205] Tried to move back new op lower bound from 7316635708763045888 to 7316635708349718528. Current Snapshot: MvccSnapshot[applied={T|T < 7316635708763045888 or (T in {7316635708763045888})}]
I20260809 15:06:54.927596 811 catalog_manager.cc:1347] Generated new cluster ID: 13fbfb64a00e4421b6847e1284ef2b5b
I20260809 15:06:54.927687 811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:06:54.973711 811 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:06:54.975742 811 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:06:54.997181 851 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 15:06:54.998908 811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3f906bb0a3433d90e3305c8e523f83: Generated new TSK 0
I20260809 15:06:55.007836 811 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 15:06:55.091639 536 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.077152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.27.186.190:44773)
0809 15:06:55.077541 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.077547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.088069 (+ 10522us) server_negotiation.cc:407] Connection header received
0809 15:06:55.089582 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.089590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.089703 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.089779 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.091357 (+ 1578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59242: BlockingRecv error: recv got EOF from 127.0.0.1:59242 (error 108)
Metrics: {"server-negotiator.queue_time_us":230}
W20260809 15:06:55.096174 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.076968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.27.186.189:40057)
0809 15:06:55.077353 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.077361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.093159 (+ 15798us) server_negotiation.cc:407] Connection header received
0809 15:06:55.094740 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.094746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.094857 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.094933 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.095899 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45898: BlockingRecv error: recv got EOF from 127.0.0.1:45898 (error 108)
Metrics: {"server-negotiator.queue_time_us":246}
W20260809 15:06:55.098565 763 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.088651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.27.186.188:39465)
0809 15:06:55.088854 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.088861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.094324 (+ 5463us) server_negotiation.cc:407] Connection header received
0809 15:06:55.096995 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.097001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.097096 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.097166 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.098289 (+ 1123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45964: BlockingRecv error: recv got EOF from 127.0.0.1:45964 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 15:06:55.104933 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.102974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.27.186.189:40057)
0809 15:06:55.103117 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.103122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.103769 (+ 647us) server_negotiation.cc:407] Connection header received
0809 15:06:55.103961 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.103964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.104057 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.104137 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.104751 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45912: BlockingRecv error: recv got EOF from 127.0.0.1:45912 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 15:06:55.111610 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.108375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.27.186.189:40057)
0809 15:06:55.108524 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.108529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.110442 (+ 1913us) server_negotiation.cc:407] Connection header received
0809 15:06:55.110601 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.110605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.110669 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.110741 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.111410 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45922: BlockingRecv error: recv got EOF from 127.0.0.1:45922 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 15:06:55.117544 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.115931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.27.186.189:40057)
0809 15:06:55.116088 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.116094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.116504 (+ 410us) server_negotiation.cc:407] Connection header received
0809 15:06:55.116651 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.116656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.116716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.116789 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.117383 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45930: BlockingRecv error: recv got EOF from 127.0.0.1:45930 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 15:06:55.125633 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.123840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.27.186.189:40057)
0809 15:06:55.124022 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.124027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.124449 (+ 422us) server_negotiation.cc:407] Connection header received
0809 15:06:55.124600 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.124605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.124700 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.124779 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.125423 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45936: BlockingRecv error: recv got EOF from 127.0.0.1:45936 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260809 15:06:55.137650 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.135986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.27.186.189:40057)
0809 15:06:55.136138 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.136144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.136596 (+ 452us) server_negotiation.cc:407] Connection header received
0809 15:06:55.136750 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.136755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.136812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.136889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.137475 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45944: BlockingRecv error: recv got EOF from 127.0.0.1:45944 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 15:06:55.158325 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.156313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.27.186.189:40057)
0809 15:06:55.156484 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.156491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.157003 (+ 512us) server_negotiation.cc:407] Connection header received
0809 15:06:55.157188 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.157192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.157270 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.157358 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.158077 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45948: BlockingRecv error: recv got EOF from 127.0.0.1:45948 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 15:06:55.195007 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.192984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.27.186.189:40057)
0809 15:06:55.193171 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.193192 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.193691 (+ 499us) server_negotiation.cc:407] Connection header received
0809 15:06:55.193869 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.193873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.193937 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.194043 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.194750 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45952: BlockingRecv error: recv got EOF from 127.0.0.1:45952 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 15:06:55.264005 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.262106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.27.186.189:40057)
0809 15:06:55.262331 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.262337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.262615 (+ 278us) server_negotiation.cc:407] Connection header received
0809 15:06:55.262796 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.262802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.262904 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.263032 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.263762 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45960: BlockingRecv error: recv got EOF from 127.0.0.1:45960 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260809 15:06:55.400970 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.399238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.27.186.189:40057)
0809 15:06:55.399470 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.399477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.399724 (+ 247us) server_negotiation.cc:407] Connection header received
0809 15:06:55.399886 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.399890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.399979 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.400064 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.400749 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45964: BlockingRecv error: recv got EOF from 127.0.0.1:45964 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 15:06:55.637820 782 catalog_manager.cc:1259] Loaded cluster ID: 13fbfb64a00e4421b6847e1284ef2b5b
I20260809 15:06:55.637917 782 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: loading cluster ID for follower catalog manager: success
I20260809 15:06:55.640403 782 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: acquiring CA information for follower catalog manager: success
I20260809 15:06:55.642120 782 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3cbb84d560f446fbe1b53737884de87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 15:06:55.662171 765 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 15:06:55.660290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.27.186.189:40057)
0809 15:06:55.660492 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:55.660499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:55.660780 (+ 281us) server_negotiation.cc:407] Connection header received
0809 15:06:55.660995 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:55.660999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:55.661071 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:55.661158 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 15:06:55.661889 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45978: BlockingRecv error: recv got EOF from 127.0.0.1:45978 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 15:06:56.001551 851 catalog_manager.cc:1259] Loaded cluster ID: 13fbfb64a00e4421b6847e1284ef2b5b
I20260809 15:06:56.001662 851 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: loading cluster ID for follower catalog manager: success
I20260809 15:06:56.004411 851 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: acquiring CA information for follower catalog manager: success
I20260809 15:06:56.006229 851 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b92c0cefd6ea44bc8845ca19228a9323: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 15:06:56.195066 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:56.187459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.27.186.189:40057)
0809 15:06:56.188149 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:56.188155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:56.188534 (+ 379us) server_negotiation.cc:407] Connection header received
0809 15:06:56.188791 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:56.188795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:56.188927 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:56.189029 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:56.189627 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.190598 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.192650 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.193130 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.193230 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:56.193847 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:56.194408 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:56.194521 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:56.194676 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 15:06:56.194741 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":348,"threads_started":1}
I20260809 15:06:56.207981 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:56.201827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.27.186.188:39465)
0809 15:06:56.202419 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:56.202423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:56.202440 (+ 17us) server_negotiation.cc:407] Connection header received
0809 15:06:56.202530 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:56.202534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:56.202665 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:56.202791 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:56.203230 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.203908 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.205759 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.206235 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.206330 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:56.206925 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:56.207355 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:56.207456 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:56.207619 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 15:06:56.207684 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":383,"threads_started":1}
I20260809 15:06:56.224392 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:56.215248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.27.186.190:44773)
0809 15:06:56.216241 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:56.216248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:56.216270 (+ 22us) server_negotiation.cc:407] Connection header received
0809 15:06:56.216383 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:56.216389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:56.216570 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:56.217342 (+ 772us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:56.217540 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.218635 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.220938 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.221590 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.222466 (+ 876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:56.223117 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:56.223484 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:56.223606 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:56.224022 (+ 416us) server_negotiation.cc:299] Negotiation successful
0809 15:06:56.224107 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":1143,"threads_started":1}
I20260809 15:06:56.228941 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 414
I20260809 15:06:56.229907 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:56.216228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.27.186.188:39465)
0809 15:06:56.216385 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:56.216390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:56.222538 (+ 6148us) server_negotiation.cc:407] Connection header received
0809 15:06:56.222785 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:56.222790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:56.222864 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:56.222924 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:56.223802 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.224767 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.227661 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.228221 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.228313 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:56.228988 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:56.229119 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:56.229183 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:56.229574 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 15:06:56.229681 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 15:06:56.229915 518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:56.230726 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 15:06:56.215023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.27.186.189:40057)
0809 15:06:56.215212 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 15:06:56.215218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 15:06:56.222302 (+ 7084us) server_negotiation.cc:407] Connection header received
0809 15:06:56.222588 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 15:06:56.222594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 15:06:56.222679 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 15:06:56.222748 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 15:06:56.223699 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.224821 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.227835 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 15:06:56.228634 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 15:06:56.228741 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 15:06:56.229613 (+ 872us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 15:06:56.229764 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 15:06:56.229837 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 15:06:56.230260 (+ 423us) server_negotiation.cc:299] Negotiation successful
0809 15:06:56.230355 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 15:06:56.367635 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 414
I20260809 15:06:56.393610 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 543
I20260809 15:06:56.394704 632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:56.496469 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 543
I20260809 15:06:56.521286 28394 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu pid 653
I20260809 15:06:56.522367 750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 15:06:56.624107 28394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvIvrqI/build/asan/bin/kudu with pid 653
2026-08-09T15:06:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4963 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9492 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356523 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST